valitydev/clickhouse-sink-connector
14
0
Fork 0
mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-06 10:35:21 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
Replicate data from MySQL, Postgres and MongoDB to ClickHouse
371 Commits 8 Branches 0 Tags 202 MiB
Python 57.3%
Java 21.6%
HTML 13.3%
ANTLR 4.8%
Shell 2.4%
Other 0.5%
8d7e85243e
Go to file
Kanthi Subramanian 8d7e85243e Fixed unit tests and pom file for maven surefile plugin
2022-09-08 13:56:21 -04:00
.github/workflows Fixed image name in github action for building docker images. 2022-05-11 21:09:38 -04:00
deploy Merge pull request #60 from Altinity/set_binlog_position_debezium 2022-08-29 08:26:01 -04:00
doc Merge pull request #60 from Altinity/set_binlog_position_debezium 2022-08-29 08:26:01 -04:00
docker More changes to k8s manifest files. 2022-08-29 07:23:07 -04:00
src/main/java/com/altinity/clickhouse/sink/connector Fixed unit tests and pom file for maven surefile plugin 2022-09-08 13:56:21 -04:00
tests Added integration tests to compare count of seed data and sysbench tables. 2022-07-22 19:01:19 -04:00
.gitignore Added unit test directory to pom.xml 2022-07-03 19:22:14 -04:00
LICENSE Initial commit 2022-03-21 11:32:45 +03:00
pom.xml Fixed unit tests and pom file for maven surefile plugin 2022-09-08 13:56:21 -04:00
README.md Updated link to main README.md 2022-08-29 17:36:40 +05:30
strimzi.yml Updated documentation with Data types mapping. 2022-04-29 13:32:53 -04:00

README.md

Altinity Sink Connector for ClickHouse

Sink connector sinks data from Kafka into Clickhouse. The connector is tested with the following converters

Features

  • Inserts, Updates and Deletes using ReplacingMergeTree/CollapsingMergeTree - Updates/Deletes
  • Deduplication logic to dedupe records from Kafka topic.(Based on Primary Key)
  • Exactly once semantics
  • Bulk insert to Clickhouse.
  • Store Kafka metadata Kafka Metadata
  • Kafka topic to ClickHouse table mapping, use case where MySQL table can be mapped to a different CH table name.
  • Store raw data in JSON(For Auditing purposes)
  • Monitoring(Using Grafana/Prometheus) Dashboard to monitor lag.
  • Kafka Offset management in ClickHouse
  • Increased Parallelism(Customize thread pool for JDBC connections)

Grafana Dashboard

Source Databases

  • MySQL (Debezium)
  • PostgreSQL (Debezium) (Testing in progress)

Quick Start

Docker image for Sink connector altinity/clickhouse-sink-connector:latest

https://hub.docker.com/r/altinity/clickhouse-sink-connector

MySQL:

cd deploy/docker
./start-docker-compose.sh

For detailed instructions Setup

Documentation