valitydev/clickhouse-sink-connector
14
0
Fork 0
mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-06 18:45:27 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
Replicate data from MySQL, Postgres and MongoDB to ClickHouse
213 Commits 8 Branches 0 Tags 202 MiB
Python 57.3%
Java 21.6%
HTML 13.3%
ANTLR 4.8%
Shell 2.4%
Other 0.5%
b0f440ab2d
Go to file
Kanthi Subramanian b0f440ab2d Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms).
2022-05-10 12:58:54 -04:00
.github/workflows Fix github runner for building docker images. 2022-05-06 10:01:35 -04:00
deploy Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms). 2022-05-10 12:58:54 -04:00
doc Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms). 2022-05-10 12:58:54 -04:00
docker Added GraalVM kafka connect image. 2022-05-08 09:28:05 -04:00
src Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms). 2022-05-10 12:58:54 -04:00
tests Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms). 2022-05-10 12:58:54 -04:00
.gitignore Added initial commit of files from the original repo 2022-03-28 09:16:31 -04:00
LICENSE Initial commit 2022-03-21 11:32:45 +03:00
pom.xml Added logic to record min and max offset of the bulk message in prometheus. 2022-05-06 09:57:30 -04:00
README.md Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms). 2022-05-10 12:58:54 -04:00
strimzi.yml Updated documentation with Data types mapping. 2022-04-29 13:32:53 -04:00

README.md

ClickHouse Sink Connector

Sink connector sinks data from Kafka into Clickhouse. The connector is tested with the following converters

  • JsonConverter

  • AvroConverter (Using Apicurio Schema Registry)

  • Currently the connector only supports Insert operations.

  • Deduplication logic to dedupe records from Kafka topic.

  • Exactly once semantics

  • Bulk insert to Clickhouse.

  • Store Kafka metadata Kafka Metadata

Documentation