mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-06 18:45:27 +00:00

Replicate data from MySQL, Postgres and MongoDB to ClickHouse

Go to file

Kanthi Subramanian 4f31af4fe3 Updated README.md to include k8s pipeline setup.		2022-06-14 13:30:01 -04:00
.github/workflows	Fixed image name in github action for building docker images.	2022-05-11 21:09:38 -04:00
deploy	Fixed logic of before/after struct. Added view as a separate table to avoid stack size exception(TOO_MUCH_RECURSION) in ClickHouse.	2022-06-14 12:55:39 -04:00
doc	Added kafka lag dashboard	2022-06-10 21:41:09 -04:00
docker	Removed sign/version column from configuration. Added test case to validate ReplacingMergeTree and CollapsingMergeTree.	2022-06-06 12:18:38 -04:00
src	Fixed logic of before/after struct. Added view as a separate table to avoid stack size exception(TOO_MUCH_RECURSION) in ClickHouse.	2022-06-14 12:55:39 -04:00
tests	Added README for explaining Updates/Deletes. Refactor logic of passing ReplacingMergeTree delete column.	2022-06-06 18:20:36 -04:00
.gitignore	Added initial commit of files from the original repo	2022-03-28 09:16:31 -04:00
LICENSE	Initial commit	2022-03-21 11:32:45 +03:00
pom.xml	Added logic to record min and max offset of the bulk message in prometheus.	2022-05-06 09:57:30 -04:00
README.md	Updated README.md to include k8s pipeline setup.	2022-06-14 13:30:01 -04:00
strimzi.yml	Updated documentation with Data types mapping.	2022-04-29 13:32:53 -04:00

Altinity Sink Connector for ClickHouse

Sink connector sinks data from Kafka into Clickhouse. The connector is tested with the following converters

Features

Inserts, Updates and Deletes using ReplacingMergeTree/CollapsingMergeTreeUpdates/Deletes
Deduplication logic to dedupe records from Kafka topic.
Exactly once semantics
Bulk insert to Clickhouse.
Store Kafka metadata Kafka Metadata