valitydev/clickhouse-sink-connector
14
0
Fork 0
mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-06 18:45:27 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
Replicate data from MySQL, Postgres and MongoDB to ClickHouse
256 Commits 8 Branches 0 Tags 202 MiB
Python 57.3%
Java 21.6%
HTML 13.3%
ANTLR 4.8%
Shell 2.4%
Other 0.5%
78634b19fa
Go to file
Kanthi Subramanian 78634b19fa Added functions to extract version and sign columns from engine column in system tables.
2022-06-03 13:15:09 -04:00
.github/workflows Fixed image name in github action for building docker images. 2022-05-11 21:09:38 -04:00
deploy Updated docker image to dockerhub for sink connector 2022-06-03 10:27:43 -04:00
doc Updated pre-requisites for local setup 2022-05-31 13:29:39 -04:00
docker make image names for readbale 2022-05-11 15:03:51 +03:00
src Added functions to extract version and sign columns from engine column in system tables. 2022-06-03 13:15:09 -04:00
tests Fixed bug with adding batch for insert statements. 2022-06-01 10:17:55 -04:00
.gitignore Added initial commit of files from the original repo 2022-03-28 09:16:31 -04:00
LICENSE Initial commit 2022-03-21 11:32:45 +03:00
pom.xml Added logic to record min and max offset of the bulk message in prometheus. 2022-05-06 09:57:30 -04:00
README.md Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms). 2022-05-10 12:58:54 -04:00
strimzi.yml Updated documentation with Data types mapping. 2022-04-29 13:32:53 -04:00

README.md

ClickHouse Sink Connector

Sink connector sinks data from Kafka into Clickhouse. The connector is tested with the following converters

  • JsonConverter

  • AvroConverter (Using Apicurio Schema Registry)

  • Currently the connector only supports Insert operations.

  • Deduplication logic to dedupe records from Kafka topic.

  • Exactly once semantics

  • Bulk insert to Clickhouse.

  • Store Kafka metadata Kafka Metadata

Documentation