mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-06 18:45:27 +00:00

Replicate data from MySQL, Postgres and MongoDB to ClickHouse

Go to file

Kanthi Subramanian a359c7ae59 Added logic to retrieve the topic partition and offset for evey block. Added unit test.		2022-06-12 15:58:59 -04:00
.github/workflows	Fixed image name in github action for building docker images.	2022-05-11 21:09:38 -04:00
deploy	Added kafka lag dashboard	2022-06-10 21:41:09 -04:00
doc	Added kafka lag dashboard	2022-06-10 21:41:09 -04:00
docker	Removed sign/version column from configuration. Added test case to validate ReplacingMergeTree and CollapsingMergeTree.	2022-06-06 12:18:38 -04:00
src	Added logic to retrieve the topic partition and offset for evey block. Added unit test.	2022-06-12 15:58:59 -04:00
tests	Added README for explaining Updates/Deletes. Refactor logic of passing ReplacingMergeTree delete column.	2022-06-06 18:20:36 -04:00
.gitignore	Added initial commit of files from the original repo	2022-03-28 09:16:31 -04:00
LICENSE	Initial commit	2022-03-21 11:32:45 +03:00
pom.xml	Added logic to record min and max offset of the bulk message in prometheus.	2022-05-06 09:57:30 -04:00
README.md	Updated README.md	2022-06-09 09:53:48 -04:00
strimzi.yml	Updated documentation with Data types mapping.	2022-04-29 13:32:53 -04:00

ClickHouse Sink Connector

Sink connector sinks data from Kafka into Clickhouse. The connector is tested with the following converters

JsonConverter
AvroConverter (Using Apicurio Schema Registry)
Inserts, Updates and Deletes using ReplacingMergeTree/CollapsingMergeTreeUpdates/Deletes
Deduplication logic to dedupe records from Kafka topic.
Exactly once semantics
Bulk insert to Clickhouse.
Store Kafka metadata Kafka Metadata

Documentation