valitydev/clickhouse-sink-connector
14
0
Fork 0
mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-06 10:35:21 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
Replicate data from MySQL, Postgres and MongoDB to ClickHouse
249 Commits 8 Branches 0 Tags 202 MiB
Python 57.3%
Java 21.6%
HTML 13.3%
ANTLR 4.8%
Shell 2.4%
Other 0.5%
b36dbac88d
Go to file
Kanthi Subramanian b36dbac88d Updated pre-requisites for local setup
2022-05-31 13:29:39 -04:00
.github/workflows Fixed image name in github action for building docker images. 2022-05-11 21:09:38 -04:00
deploy Changed the sign column to a Sink configuration variable. 2022-05-26 16:29:26 -04:00
doc Updated pre-requisites for local setup 2022-05-31 13:29:39 -04:00
docker make image names for readbale 2022-05-11 15:03:51 +03:00
src Refactoring to support retrieving before/after columns and logic to insert records(before and after separately) when its an update. 2022-05-31 11:18:14 -04:00
tests Changed the sign column to a Sink configuration variable. 2022-05-26 16:29:26 -04:00
.gitignore Added initial commit of files from the original repo 2022-03-28 09:16:31 -04:00
LICENSE Initial commit 2022-03-21 11:32:45 +03:00
pom.xml Added logic to record min and max offset of the bulk message in prometheus. 2022-05-06 09:57:30 -04:00
README.md Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms). 2022-05-10 12:58:54 -04:00
strimzi.yml Updated documentation with Data types mapping. 2022-04-29 13:32:53 -04:00

README.md

ClickHouse Sink Connector

Sink connector sinks data from Kafka into Clickhouse. The connector is tested with the following converters

  • JsonConverter

  • AvroConverter (Using Apicurio Schema Registry)

  • Currently the connector only supports Insert operations.

  • Deduplication logic to dedupe records from Kafka topic.

  • Exactly once semantics

  • Bulk insert to Clickhouse.

  • Store Kafka metadata Kafka Metadata

Documentation