valitydev/clickhouse-sink-connector
14
0
Fork 0
mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-06 18:45:27 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
371 Commits 8 Branches 0 Tags 202 MiB
8d7e85243e
Commit Graph

62 Commits

Author SHA1 Message Date
Kanthi
fd3038e33a
Merge pull request #60 from Altinity/set_binlog_position_debezium 
Manually set  binlog position before starting streaming.
 2022-08-29 08:26:01 -04:00
Kanthi Subramanian
7002bcd522 Update debezium setup instructions. 2022-08-29 17:34:42 +05:30
Kanthi Subramanian
3feb609454 Instructions to set the binlog position manually before starting ingestion. 2022-08-29 13:35:23 +05:30
Kanthi Subramanian
0a1567a4b4 Instructions to set the binlog position manually before starting ingestion. 2022-08-27 17:27:11 +05:30
Kanthi Subramanian
2e13fd4964 Added kubernetes pod for debezium. 2022-08-25 08:57:24 -04:00
Kanthi Subramanian
1a0cabffcb Added redpanda UI(console), Strimzi debezium image, Added confluent schema registry ks8 manifest. 2022-08-24 02:09:44 -04:00
Kanthi Subramanian
f2d4be6277 Added redpanda 3 node cluster configuration and schema registry confluent. 2022-08-22 06:45:13 -04:00
Kanthi Subramanian
a4f605e886 Renamed docker compose file to follow the override format. Added configuration changes to connect to external MySQL Database. 2022-08-18 01:59:12 -04:00
Kanthi Subramanian
3a9f800c96 Cleanup docker compose files to remove duplicates and use the docker override syntax 2022-08-13 18:30:11 +05:30
Kanthi Subramanian
edef4968a8 Fixed logic of auto create table for empty primary key. 
Load ontime airline dataset.
 2022-08-11 09:54:06 -04:00
Kanthi Subramanian
08f9806916 Updated README instructions. 2022-08-02 17:46:37 -04:00
Kanthi Subramanian
0f851a1996 Fixed logic of retry for auto creating table 2022-08-02 17:41:38 -04:00
Kanthi Subramanian
9c6b376667 Added partition to offset metrics(Prometheus) 2022-07-27 11:01:52 -04:00
Kanthi Subramanian
ab42714f88 Added a map of topics to DBWriter instances to cover scenario where messages of multiple topics will be sent to the same Kafka SinkTask. 
Fixed bug with extra addBatch
Added logic to check for indexes before inserting additional metadata
Added logic to not remove records if the addBatch was unsuccessful.
 2022-07-19 17:41:52 -04:00
Kanthi Subramanian
3bd0c4fb42 Move flush timeout as a configuration variable. Change docker-compose to use environment variable for sink docker version. 2022-07-19 14:07:02 -04:00
Kanthi Subramanian
4ee8667b69 Added logic to flush records based on buffer size and number of records. 2022-07-15 16:23:19 -05:00
Kanthi Subramanian
c436a14198 Added functionality to create auto tables for multiple primary key columns 2022-07-13 16:25:16 -05:00
Kanthi Subramanian
ba9f44e181 Enable GTID in MySQL docker image, added logic to set the version in ReplacingMergeTree to GTID if available, fallback to timestamp msecs(from debezium). 
Fixed auto create tables for multiple primary keys.
 2022-07-13 14:01:15 -05:00
Kanthi Subramanian
831fb3332f Added class to store data type mapping from kafka connect schema to Clickhouse schema, prep work for auto create tables. 2022-06-22 20:47:33 -04:00
Kanthi Subramanian
255a0e8bd3 Changed thread pool size to a configurable parameter, Changed the logic of removing records from the shared records queue. 2022-06-21 16:59:33 -04:00
Kanthi Subramanian
6db50db39c Added documentation and sysbench read/write script. 2022-06-20 14:24:33 -04:00
Kanthi Subramanian
45dde57b8b Added document for SysBench setup. 2022-06-20 11:52:55 -04:00
Kanthi Subramanian
c6b79b6a1a Added kafka lag dashboard 2022-06-10 21:41:09 -04:00
Kanthi Subramanian
cc9742f538 Added README for explaining Updates/Deletes. Refactor logic of passing ReplacingMergeTree delete column. 2022-06-06 18:20:36 -04:00
Kanthi Subramanian
e1334e94ce Added README for explaining Updates/Deletes. 2022-06-06 13:14:04 -04:00
Kanthi Subramanian
b36dbac88d Updated pre-requisites for local setup 2022-05-31 13:29:39 -04:00
Kanthi Subramanian
690334f3a7 Changed the sign column to a Sink configuration variable. 2022-05-26 16:29:26 -04:00
Kanthi Subramanian
0d3a29dbc3 Updated Local setup instructions. 2022-05-24 09:50:03 -04:00
Vladislav Klimenko
dd01674b14 extract kubectl finalization 2022-05-12 17:20:16 +03:00
Vladislav Klimenko
6e76e34172 make port forward as a background process 2022-05-12 17:18:35 +03:00
Vladislav Klimenko
29f2454106 polish pipeline 2022-05-12 17:09:38 +03:00
Vladislav Klimenko
ebb0af59b9 adjust port forwarder 2022-05-12 17:06:20 +03:00
Vladislav Klimenko
7e2c18f134 extract port forward 2022-05-12 16:57:17 +03:00
Vladislav Klimenko
d2c8f37270 load to local registry 2022-05-12 16:57:05 +03:00
Vladislav Klimenko
d059502963 docs: do not pull altinity images yet 2022-05-12 11:17:10 +03:00
Kanthi Subramanian
63252d82d4 Enable transactions metadata support in config. 2022-05-10 17:54:28 -04:00
Kanthi Subramanian
ff71203377 Fixed bug with kafka metadata columns not added when there is a source, CH columns mismatch. Added unittest for QueryFormatter class. 2022-05-10 15:59:53 -04:00
Kanthi Subramanian
b0f440ab2d Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms). 2022-05-10 12:58:54 -04:00
Kanthi Subramanian
7648df357d Added test data for all blob types. Changed input function query to support cases when there are more MySQL columns compared to Clickhouse columns. 2022-05-09 13:56:11 -04:00
Kanthi Subramanian
bf8604d04b Added task id to prometheus metrics. 2022-05-07 17:20:15 -04:00
Vladislav Klimenko
2b01571692 split commands for copy+paste 2022-05-06 20:39:59 +03:00
Vladislav Klimenko
ff4d1043d1 redpanda condition 2022-05-06 15:38:01 +03:00
Vladislav Klimenko
5d59cdb4db doc: move k8s docs to main folder 2022-05-06 15:14:20 +03:00
Kanthi Subramanian
e1cf390123 Added unit test case and logic to limit the Date values to clickhouse supported date ranges. 2022-05-05 11:09:05 -04:00
Kanthi Subramanian
2e5627611a Updated Sink configuration documentation. 2022-05-04 14:52:27 -04:00
Kanthi Subramanian
315d147296 Sink connector configuration documentation. 2022-04-29 21:00:54 -04:00
Kanthi Subramanian
0ce41e763d Updated documentation with Data types mapping. 2022-04-29 13:32:53 -04:00
Kanthi Subramanian
67a493c629 Updated documentation. 2022-04-28 17:20:19 -04:00
Kanthi Subramanian
a1e1e3fe66 Change flush time to a kafka connect configuration variable. 2022-04-25 12:04:44 -04:00
Kanthi Subramanian
41dce242c5 Enabled JMX in sink , added jmx-prometheus exporter, prometheus and grafana docker images. 2022-04-07 16:41:45 -04:00