Kanthi Subramanian
a4f605e886
Renamed docker compose file to follow the override format. Added configuration changes to connect to external MySQL Database.
2022-08-18 01:59:12 -04:00
Kanthi Subramanian
3a9f800c96
Cleanup docker compose files to remove duplicates and use the docker override syntax
2022-08-13 18:30:11 +05:30
Kanthi Subramanian
edef4968a8
Fixed logic of auto create table for empty primary key.
...
Load ontime airline dataset.
2022-08-11 09:54:06 -04:00
Kanthi Subramanian
08f9806916
Updated README instructions.
2022-08-02 17:46:37 -04:00
Kanthi Subramanian
0f851a1996
Fixed logic of retry for auto creating table
2022-08-02 17:41:38 -04:00
Kanthi Subramanian
9c6b376667
Added partition to offset metrics(Prometheus)
2022-07-27 11:01:52 -04:00
Kanthi Subramanian
ab42714f88
Added a map of topics to DBWriter instances to cover scenario where messages of multiple topics will be sent to the same Kafka SinkTask.
...
Fixed bug with extra addBatch
Added logic to check for indexes before inserting additional metadata
Added logic to not remove records if the addBatch was unsuccessful.
2022-07-19 17:41:52 -04:00
Kanthi Subramanian
3bd0c4fb42
Move flush timeout as a configuration variable. Change docker-compose to use environment variable for sink docker version.
2022-07-19 14:07:02 -04:00
Kanthi Subramanian
4ee8667b69
Added logic to flush records based on buffer size and number of records.
2022-07-15 16:23:19 -05:00
Kanthi Subramanian
c436a14198
Added functionality to create auto tables for multiple primary key columns
2022-07-13 16:25:16 -05:00
Kanthi Subramanian
ba9f44e181
Enable GTID in MySQL docker image, added logic to set the version in ReplacingMergeTree to GTID if available, fallback to timestamp msecs(from debezium).
...
Fixed auto create tables for multiple primary keys.
2022-07-13 14:01:15 -05:00
Kanthi Subramanian
831fb3332f
Added class to store data type mapping from kafka connect schema to Clickhouse schema, prep work for auto create tables.
2022-06-22 20:47:33 -04:00
Kanthi Subramanian
255a0e8bd3
Changed thread pool size to a configurable parameter, Changed the logic of removing records from the shared records queue.
2022-06-21 16:59:33 -04:00
Kanthi Subramanian
6db50db39c
Added documentation and sysbench read/write script.
2022-06-20 14:24:33 -04:00
Kanthi Subramanian
45dde57b8b
Added document for SysBench setup.
2022-06-20 11:52:55 -04:00
Kanthi Subramanian
c6b79b6a1a
Added kafka lag dashboard
2022-06-10 21:41:09 -04:00
Kanthi Subramanian
cc9742f538
Added README for explaining Updates/Deletes. Refactor logic of passing ReplacingMergeTree delete column.
2022-06-06 18:20:36 -04:00
Kanthi Subramanian
e1334e94ce
Added README for explaining Updates/Deletes.
2022-06-06 13:14:04 -04:00
Kanthi Subramanian
b36dbac88d
Updated pre-requisites for local setup
2022-05-31 13:29:39 -04:00
Kanthi Subramanian
690334f3a7
Changed the sign column to a Sink configuration variable.
2022-05-26 16:29:26 -04:00
Kanthi Subramanian
0d3a29dbc3
Updated Local setup instructions.
2022-05-24 09:50:03 -04:00
Vladislav Klimenko
dd01674b14
extract kubectl finalization
2022-05-12 17:20:16 +03:00
Vladislav Klimenko
6e76e34172
make port forward as a background process
2022-05-12 17:18:35 +03:00
Vladislav Klimenko
29f2454106
polish pipeline
2022-05-12 17:09:38 +03:00
Vladislav Klimenko
ebb0af59b9
adjust port forwarder
2022-05-12 17:06:20 +03:00
Vladislav Klimenko
7e2c18f134
extract port forward
2022-05-12 16:57:17 +03:00
Vladislav Klimenko
d2c8f37270
load to local registry
2022-05-12 16:57:05 +03:00
Vladislav Klimenko
d059502963
docs: do not pull altinity images yet
2022-05-12 11:17:10 +03:00
Kanthi Subramanian
63252d82d4
Enable transactions metadata support in config.
2022-05-10 17:54:28 -04:00
Kanthi Subramanian
ff71203377
Fixed bug with kafka metadata columns not added when there is a source, CH columns mismatch. Added unittest for QueryFormatter class.
2022-05-10 15:59:53 -04:00
Kanthi Subramanian
b0f440ab2d
Added more source kafka metadata columns(binlog pos, server id, thread, source ts_ms).
2022-05-10 12:58:54 -04:00
Kanthi Subramanian
7648df357d
Added test data for all blob types. Changed input function query to support cases when there are more MySQL columns compared to Clickhouse columns.
2022-05-09 13:56:11 -04:00
Kanthi Subramanian
bf8604d04b
Added task id to prometheus metrics.
2022-05-07 17:20:15 -04:00
Vladislav Klimenko
2b01571692
split commands for copy+paste
2022-05-06 20:39:59 +03:00
Vladislav Klimenko
ff4d1043d1
redpanda condition
2022-05-06 15:38:01 +03:00
Vladislav Klimenko
5d59cdb4db
doc: move k8s docs to main folder
2022-05-06 15:14:20 +03:00
Kanthi Subramanian
e1cf390123
Added unit test case and logic to limit the Date values to clickhouse supported date ranges.
2022-05-05 11:09:05 -04:00
Kanthi Subramanian
2e5627611a
Updated Sink configuration documentation.
2022-05-04 14:52:27 -04:00
Kanthi Subramanian
315d147296
Sink connector configuration documentation.
2022-04-29 21:00:54 -04:00
Kanthi Subramanian
0ce41e763d
Updated documentation with Data types mapping.
2022-04-29 13:32:53 -04:00
Kanthi Subramanian
67a493c629
Updated documentation.
2022-04-28 17:20:19 -04:00
Kanthi Subramanian
a1e1e3fe66
Change flush time to a kafka connect configuration variable.
2022-04-25 12:04:44 -04:00
Kanthi Subramanian
41dce242c5
Enabled JMX in sink , added jmx-prometheus exporter, prometheus and grafana docker images.
2022-04-07 16:41:45 -04:00
Kanthi Subramanian
3dc71af131
Added support for creating clickhouse connection from kafka connect sink config variables.
2022-04-05 16:23:25 -04:00
Kanthi Subramanian
9a5db02a49
Added support for remote debugging Kafka connector and entrypoint db to load data to msysql and clickhouse.
2022-04-05 11:03:49 -04:00
Vladislav Klimenko
245d80ebb6
doc: glosasry
2022-04-04 12:28:35 +03:00
Vladislav Klimenko
c8bf0b308c
doc: index
2022-04-04 12:27:21 +03:00
Vladislav Klimenko
92c232811b
doc: pipeline clarification
2022-04-04 12:18:18 +03:00
Vladislav Klimenko
d5893bd291
doc: setup
2022-04-04 11:06:17 +03:00
Vladislav Klimenko
724aaa2877
env: readme for img
2022-04-04 11:05:49 +03:00
