2022-06-13 18:35:34 +00:00
|
|
|
# Altinity Sink Connector for ClickHouse
|
2022-04-01 12:00:03 +00:00
|
|
|
|
2022-04-29 17:32:53 +00:00
|
|
|
Sink connector sinks data from Kafka into Clickhouse.
|
|
|
|
|
The connector is tested with the following converters
|
|
|
|
|
- JsonConverter
|
2022-08-09 22:29:25 +00:00
|
|
|
- AvroConverter (Using [Apicurio Schema Registry](https://www.apicur.io/registry/) or Confluent Schema Registry)
|
2022-04-29 17:32:53 +00:00
|
|
|
|
2022-08-02 21:41:38 +00:00
|
|
|
|
2022-06-14 17:30:01 +00:00
|
|
|
# Features
|
2022-06-29 19:48:01 +00:00
|
|
|
- Inserts, Updates and Deletes using ReplacingMergeTree/CollapsingMergeTree - [Updates/Deletes](doc/mutable_data.md)
|
|
|
|
|
- Deduplication logic to dedupe records from Kafka topic.(Based on Primary Key)
|
2022-05-06 16:25:33 +00:00
|
|
|
- Exactly once semantics
|
2022-06-29 19:48:01 +00:00
|
|
|
- Bulk insert to Clickhouse.
|
2022-05-10 16:58:54 +00:00
|
|
|
- Store Kafka metadata [Kafka Metadata](doc/Kafka_metadata.md)
|
2022-06-29 19:48:01 +00:00
|
|
|
- Kafka topic to ClickHouse table mapping, use case where MySQL table can be mapped to a different CH table name.
|
|
|
|
|
- Store raw data in JSON(For Auditing purposes)
|
|
|
|
|
- Monitoring(Using Grafana/Prometheus) Dashboard to monitor lag.
|
|
|
|
|
- Kafka Offset management in ClickHouse
|
|
|
|
|
- Increased Parallelism(Customize thread pool for JDBC connections)
|
2022-04-01 12:00:03 +00:00
|
|
|
|
2022-08-02 21:41:38 +00:00
|
|
|
### Grafana Dashboard
|
2022-08-13 13:00:11 +00:00
|
|
|
|
|
|
|
|
|
2022-06-14 17:30:01 +00:00
|
|
|
# Source Databases
|
|
|
|
|
- MySQL (Debezium)
|
2022-06-16 21:29:04 +00:00
|
|
|
- PostgreSQL (Debezium) (Testing in progress)
|
2022-04-29 17:32:53 +00:00
|
|
|
|
2022-08-13 13:00:11 +00:00
|
|
|
### Quick Start
|
|
|
|
|
Docker image for Sink connector `altinity/clickhouse-sink-connector:latest`
|
|
|
|
|
|
|
|
|
|
https://hub.docker.com/r/altinity/clickhouse-sink-connector
|
|
|
|
|
|
|
|
|
|
### MySQL:
|
|
|
|
|
```bash
|
|
|
|
|
cd deploy/docker
|
|
|
|
|
./start-docker-compose.sh
|
|
|
|
|
```
|
|
|
|
|
For detailed instructions [Setup](doc/setup.md)
|
2022-04-29 17:32:53 +00:00
|
|
|
## Documentation
|
|
|
|
|
- [Data Types](doc/DataTypes.md)
|
|
|
|
|
- [Architecture](doc/architecture.md)
|
2022-06-14 17:30:01 +00:00
|
|
|
- [Local Setup - Docker Compose](doc/setup.md)
|
2022-08-29 12:06:40 +00:00
|
|
|
- [Debezium Setup](doc/debezium_setup.md)
|
2022-06-14 17:30:01 +00:00
|
|
|
- [Kubernetes Setup](doc/k8s_pipeline_setup.md)
|
2022-04-29 17:32:53 +00:00
|
|
|
- [Sink Configuration](doc/sink_configuration.md)
|
2022-06-20 18:24:33 +00:00
|
|
|
- [Testing](doc/TESTING.md)
|
2022-08-09 22:29:25 +00:00
|
|
|
- [Performance Benchmarking](doc/Performance.md)
|
