46 Commits

Author	SHA1	Message	Date
Kanthi Subramanian	53f53ea814	Fixed comparison of Kafka metadata column name.	2022-04-12 14:07:30 -04:00
Kanthi Subramanian	c70183f37d	Store kafka metadata columns as Enum.	2022-04-12 13:28:29 -04:00
Kanthi Subramanian	d9fe27e6dc	Fixed merge conflict	2022-04-12 09:57:15 -04:00
Kanthi Subramanian	d3ff440f66	Merge remote-tracking branch 'origin/main' ... # Conflicts: # src/main/java/com/altinity/clickhouse/sink/connector/ClickHouseSinkTask.java	2022-04-12 09:48:55 -04:00
Kanthi Subramanian	19555ea05b	Added logic to persist kafka metadata to Clickhouse(topic, partition, offset and timestamp)	2022-04-12 09:47:14 -04:00
Vladislav Klimenko	a5edad2456	fix localhost typo	2022-04-12 16:40:10 +03:00
Vladislav Klimenko	cb32d36199	accept buffer size as a parameter	2022-04-12 16:35:52 +03:00
Vladislav Klimenko	08af306400	pass config to deduplicator	2022-04-12 16:35:26 +03:00
Vladislav Klimenko	c20818bc3f	introduce task_id	2022-04-12 16:34:46 +03:00
Vladislav Klimenko	ef7e058139	move TASK_ID as a config variable	2022-04-12 16:34:21 +03:00
Vladislav Klimenko	02e52953f3	formatter	2022-04-12 12:34:21 +03:00
Kanthi Subramanian	e19ce4dc97	Fixed Delete script(bash) , Added logic to handle mismatch in column schemas.	2022-04-11 09:09:05 -04:00
Kanthi Subramanian	d74013849d	Merge remote-tracking branch 'origin/main'	2022-04-11 08:45:18 -04:00
Kanthi Subramanian	9cd3bb9b33	Updated clickhouse schema.	2022-04-11 08:45:13 -04:00
Vladislav Klimenko	d9c187b750	Merge remote-tracking branch 'altinity/main' into main	2022-04-11 12:54:54 +03:00
Vladislav Klimenko	f5665e42d1	dev: simplify unused code	2022-04-11 12:54:06 +03:00
Kanthi Subramanian	48de47adee	Replaced HashMap to LinkedHashMap to preserve insertion order. Updated JUnit to 5.	2022-04-10 12:25:13 -04:00
Kanthi Subramanian	9814555de0	Added logic to use input function for preparedStatement - bulk Insert	2022-04-09 15:14:43 -04:00
Kanthi Subramanian	b55cda0ff5	Added support for more datatypes(DateTime, Time, TinyInt, BigInt)	2022-04-08 15:42:17 -04:00
Kanthi Subramanian	a2aaef1ef6	Refactor ClickHouseConverter convert function.	2022-04-07 17:33:37 -04:00
Kanthi Subramanian	ac5e468927	Added test case to test bulk insert and rollback.	2022-04-07 13:07:34 -04:00
Kanthi Subramanian	740071a842	Fixed convertRecord() so that SinkRecords are converted to Struct. Fixed debezium and Sink configuration variables.	2022-04-06 09:52:58 -04:00
Kanthi Subramanian	c979f93b8e	Fixed clickhouse host name and unit tests.	2022-04-05 17:55:08 -04:00
Kanthi Subramanian	3dc71af131	Added support for creating clickhouse connection from kafka connect sink config variables.	2022-04-05 16:23:25 -04:00
Kanthi Subramanian	82092270f1	Merge remote-tracking branch 'origin/main'	2022-04-05 11:03:57 -04:00
Kanthi Subramanian	9a5db02a49	Added support for remote debugging Kafka connector and entrypoint db to load data to msysql and clickhouse.	2022-04-05 11:03:49 -04:00
Vladislav Klimenko	fb91365c4e	dev: dedup based on key map and queue	2022-04-05 17:15:49 +03:00
Vladislav Klimenko	fdc80e8e92	dev: dedup	2022-04-05 12:27:11 +03:00
Vladislav Klimenko	89048000d4	dev: use logger everywhere, simplify code	2022-04-05 01:55:05 +03:00
Vladislav Klimenko	edb6cf93e3	dev: merge	2022-04-04 18:18:07 +03:00
Vladislav Klimenko	99c479f1f3	dev: workflow	2022-04-04 17:41:07 +03:00
Kanthi Subramanian	2a0431614d	Added logic to create a new thread to handle flushing of records to Clickhouse	2022-04-04 09:41:54 -04:00
Vladislav Klimenko	a11442bc4a	env: move config files	2022-04-04 11:34:53 +03:00
Kanthi Subramanian	5077fe1efa	Added logic to use Clickhouse JDBC connector, convert the debezium data types and perform a batch insert.	2022-04-01 10:19:15 -04:00
Vladislav Klimenko	95ec8e1401	adoption	2022-04-01 14:16:13 +03:00
Vladislav Klimenko	ba49e3642f	formatter	2022-04-01 12:34:21 +03:00
Vladislav Klimenko	12b1f95b24	dev: validators	2022-04-01 12:28:16 +03:00
Vladislav Klimenko	d68cd9d5ec	clarify task	2022-04-01 12:09:52 +03:00
Vladislav Klimenko	643b092193	preCommit	2022-03-30 17:59:21 +03:00
Vladislav Klimenko	cadf0a2a15	dev: group connector configuration variables	2022-03-30 17:51:00 +03:00
Vladislav Klimenko	7ec8056e4e	dev: clarify connector	2022-03-30 15:25:30 +03:00
Vladislav Klimenko	f465e6e311	dev: merge conflicts	2022-03-30 15:12:50 +03:00
Kanthi Subramanian	40a781d3a9	Added initial commit of files from the original repo	2022-03-28 09:16:31 -04:00
Vladislav Klimenko	0a8e5c21e1	clean and log errors	2022-03-25 16:37:05 +03:00
Vladislav Klimenko	87a5f3d22a	package ClickHOuse client	2022-03-25 15:57:32 +03:00
Vladislav Klimenko	07a655491e	dev: start	2022-03-22 17:18:10 +03:00