valitydev/clickhouse-sink-connector
14
0
Fork 0
mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-06 10:35:21 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
337 Commits 8 Branches 0 Tags 202 MiB
edef4968a8
Commit Graph

337 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kanthi Subramanian
edef4968a8 Fixed logic of auto create table for empty primary key. 
Load ontime airline dataset.
 2022-08-11 09:54:06 -04:00
Kanthi
ed042345e0
Merge pull request #37 from Altinity/docker-compose-podman-changes 
Removed links and replace with depends_on. Fix SINK_VERSION error.
 2022-08-10 21:08:24 -04:00
Kanthi Subramanian
7d9eb39453 Removed links and replace with depends_on. Fix SINK_VERSION error. 2022-08-10 21:07:29 -04:00
Kanthi
4343f6a553
Merge pull request #31 from Altinity/21-debezium-lag-milliseconds-behind-source 
Add max.batch.size and max.queue.size to debezium configuration to im…
 2022-08-10 03:04:49 +05:30
Kanthi Subramanian
64b89f6aa7 Add max.batch.size and max.queue.size to debezium configuration to improve lag. 2022-08-09 17:33:13 -04:00
Kanthi
d6c698bda1
Merge pull request #29 from Altinity/fix_auto_create_table_retry 
Fix auto create table retry
 2022-08-02 17:49:43 -04:00
Kanthi Subramanian
08f9806916 Updated README instructions. 2022-08-02 17:46:37 -04:00
Kanthi Subramanian
0f851a1996 Fixed logic of retry for auto creating table 2022-08-02 17:41:38 -04:00
Kanthi
5a7b659af2
Merge pull request #28 from Altinity/json_datatype_support 
Support for JSON data type.
 2022-08-02 12:33:46 -04:00
Kanthi Subramanian
074d16b5d0 Support for JSON data type. 
Both MySQL/Postgres JSON types map to String with io.debezium.data.JSON name
 2022-08-02 10:51:13 -04:00
Kanthi
cec388544a
Merge pull request #26 from Altinity/fix_grafana_dashboard_loading 
Fixed grafana dashboard auto loading.
 2022-08-01 15:29:29 -04:00
Kanthi Subramanian
1d3e584058 Fixed grafana dashboard auto loading. 2022-08-01 15:09:53 -04:00
Kanthi
43d2ea85bb
Merge pull request #25 from Altinity/revert-23-15-mysql-bigint-to-ch-int64-doesnt-work-correct-with-autocreatetables-true-setting 
Revert 23 15 mysql bigint to ch int64 doesnt work correct with autocreatetables true setting
 2022-07-31 16:44:43 -04:00
Kanthi
a30686c34a
Revert "Map INT32 Kafka connect to INT256 CH type" 2022-07-31 16:44:28 -04:00
Kanthi
4436bd2970
Merge pull request #24 from Altinity/confluent_schema_registry 
Confluent schema registry
 2022-07-31 10:47:28 -04:00
Kanthi
34d2060bf8
Merge branch 'develop' into confluent_schema_registry 2022-07-31 10:47:21 -04:00
Kanthi
f5537d0953
Merge pull request #23 from Altinity/15-mysql-bigint-to-ch-int64-doesnt-work-correct-with-autocreatetables-true-setting 
Map INT32 Kafka connect to INT256 CH type
 2022-07-31 10:35:40 -04:00
Kanthi Subramanian
f13f0c9836 Confluent schema registry changes. 2022-07-31 10:34:50 -04:00
Kanthi Subramanian
cc5cfb6407 Map INT32 Kafka connect to INT256 CH type 2022-07-31 10:05:23 -04:00
Kanthi
5a65eee722
Merge pull request #19 from Altinity/debezium_monitoring_grafana 
Added steps to create grafana docker image for debezium connector and…
 2022-07-29 10:48:57 -04:00
Kanthi Subramanian
9c6b376667 Added partition to offset metrics(Prometheus) 2022-07-27 11:01:52 -04:00
Kanthi Subramanian
7572f29f9e Added logic to persist metrics and sink connector grafana dashboard. 2022-07-26 17:06:05 -04:00
Kanthi Subramanian
6ce8c4349d Added steps to create grafana docker image for debezium connector and add dashboard, datasources. 2022-07-25 15:41:47 -04:00
Kanthi
a1ec890bad
Merge pull request #17 from Altinity/integration_tests_check_count 
Integration tests check count
 2022-07-22 19:03:16 -04:00
Kanthi
f67a92e260
Merge pull request #16 from Altinity/create_sysbench_script_combine_tests 
Create sysbench script combine tests
 2022-07-22 19:02:59 -04:00
Kanthi
99e12f93e2
Merge pull request #14 from Altinity/fix_thread_flush_logic 
Refactor logic to buffer records and flush.
 2022-07-22 19:02:11 -04:00
Kanthi Subramanian
17604d1ab6 Added integration tests to compare count of seed data and sysbench tables. 2022-07-22 19:01:19 -04:00
Kanthi Subramanian
2839b960c2 Combine all the sysbench tests to a single bash script. 2022-07-22 12:18:42 -04:00
Kanthi Subramanian
0d24b902e3 Refactor logic to buffer records and flush. 2022-07-21 19:27:00 -04:00
Kanthi Subramanian
ab42714f88 Added a map of topics to DBWriter instances to cover scenario where messages of multiple topics will be sent to the same Kafka SinkTask. 
Fixed bug with extra addBatch
Added logic to check for indexes before inserting additional metadata
Added logic to not remove records if the addBatch was unsuccessful.
 2022-07-19 17:41:52 -04:00
Kanthi
7e822f13c7
Merge pull request #10 from aadant/fix_unix_permissions 
Fix unix permissions
 2022-07-19 16:08:21 -04:00
Arnaud Adant
31595e3496 Fix unix permissions 2022-07-19 14:54:31 -05:00
Kanthi Subramanian
3bd0c4fb42 Move flush timeout as a configuration variable. Change docker-compose to use environment variable for sink docker version. 2022-07-19 14:07:02 -04:00
Kanthi Subramanian
4ee8667b69 Added logic to flush records based on buffer size and number of records. 2022-07-15 16:23:19 -05:00
Kanthi Subramanian
601e900fab Fixes to include decimal scale and precision when auto creating CH tables. 2022-07-14 14:46:35 -05:00
Kanthi Subramanian
5dfdf7beb4 Fixed bug with schema inference when auto creating tables, Write kafka offset data only when config variable is set to true. 2022-07-14 09:53:28 -05:00
Kanthi Subramanian
c436a14198 Added functionality to create auto tables for multiple primary key columns 2022-07-13 16:25:16 -05:00
Kanthi Subramanian
ba9f44e181 Enable GTID in MySQL docker image, added logic to set the version in ReplacingMergeTree to GTID if available, fallback to timestamp msecs(from debezium). 
Fixed auto create tables for multiple primary keys.
 2022-07-13 14:01:15 -05:00
Kanthi Subramanian
201ffc46bb Refactoring, use a column to index map so that the Query template and the logic of adding to batch are preformed in the same order(JDBC bulk insert). 2022-07-12 11:18:56 -05:00
Kanthi Subramanian
fa660f6b4f Fixed bugs in alter table and create table. 2022-07-10 14:55:15 -04:00
Kanthi Subramanian
e58474aacd Changes to support alter table and create table. 2022-07-08 23:38:40 -04:00
Kanthi Subramanian
142f6bd76a Changes to support alter table and create table. 2022-07-08 21:51:04 -04:00
Kanthi Subramanian
6c1f24524b Added logic to extract primary key from Kafka connect key, used for alter table. 2022-07-07 22:18:26 -04:00
Kanthi Subramanian
28fc66213d Added unit tests for data type mapper. 2022-07-06 08:13:08 -04:00
Kanthi Subramanian
d623cb5937 Added unit test directory to pom.xml 2022-07-03 19:22:14 -04:00
Kanthi Subramanian
994dc54113 Updated list of features. 2022-06-29 15:48:01 -04:00
Kanthi Subramanian
6f49271d78 More data types mapping for auto create tables. 2022-06-29 15:43:07 -04:00
Kanthi Subramanian
5f07dcdd28 Added more test cases for data type mapper. 2022-06-28 22:27:59 -04:00
Kanthi Subramanian
77daad80a6 Added more test cases for autocreate table, removed tableName passed to DbWriter.Fixed warnings 2022-06-27 12:18:01 -04:00
Kanthi Subramanian
d3a51d8ea5 Added test case for auto create table feature. 2022-06-27 11:23:32 -04:00