valitydev/clickhouse-sink-connector
14
0
Fork 0
mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-07 02:55:23 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
5c9c1cd55c
clickhouse-sink-connector/doc/setup.md
Vladislav Klimenko 92c232811b doc: pipeline clarification
2022-04-04 12:18:18 +03:00

1.3 KiB
Raw Blame History

This doc describes how to set up CDC pipeline

Pipeline

pipeline

Setup local pipeline

docker-compose

Full pipeline can be launched via docker-compose with the help of docker-compose.yaml It will start:

  1. MySQL
  2. Zookeeper
  3. Debezium MySQL connector
  4. RedPanda
  5. clickhouse-kafka-sink-connector
  6. Clickhouse

In order to launch clickhouse-kafka-sink-connector appropariate docker image is required, which can be built as described in Image

cd deploy/docker
docker-compose up

Image

Docker image can be created with provided Dockerfile and build script

ToDO: Create Kafka connector image with Mysql Create JAR file by running the following command and copy to the /libs directory of Kafka.

mvn install

Copy MYSQL libs from container/libs to the kafka libs directory.

Start the Kafka connect process and pass the properties file for both MYSQL and Clickhouse properties.

./connect-standalone.sh ../config/connect-standalone.properties ../../kafka-connect-clickhouse/kcch-connector/src/main/config/mysql-debezium.properties ../../kafka-connect-clickhouse/kcch-connector/src/main/config/clickhouse-sink.properties