Kanthi Subramanian
6ce8c4349d
Added steps to create grafana docker image for debezium connector and add dashboard, datasources.
2022-07-25 15:41:47 -04:00
Kanthi
a1ec890bad
Merge pull request #17 from Altinity/integration_tests_check_count
...
Integration tests check count
2022-07-22 19:03:16 -04:00
Kanthi
f67a92e260
Merge pull request #16 from Altinity/create_sysbench_script_combine_tests
...
Create sysbench script combine tests
2022-07-22 19:02:59 -04:00
Kanthi
99e12f93e2
Merge pull request #14 from Altinity/fix_thread_flush_logic
...
Refactor logic to buffer records and flush.
2022-07-22 19:02:11 -04:00
Kanthi Subramanian
17604d1ab6
Added integration tests to compare count of seed data and sysbench tables.
2022-07-22 19:01:19 -04:00
Kanthi Subramanian
2839b960c2
Combine all the sysbench tests to a single bash script.
2022-07-22 12:18:42 -04:00
Kanthi Subramanian
0d24b902e3
Refactor logic to buffer records and flush.
2022-07-21 19:27:00 -04:00
Kanthi Subramanian
ab42714f88
Added a map of topics to DBWriter instances to cover scenario where messages of multiple topics will be sent to the same Kafka SinkTask.
...
Fixed bug with extra addBatch
Added logic to check for indexes before inserting additional metadata
Added logic to not remove records if the addBatch was unsuccessful.
2022-07-19 17:41:52 -04:00
Kanthi
7e822f13c7
Merge pull request #10 from aadant/fix_unix_permissions
...
Fix unix permissions
2022-07-19 16:08:21 -04:00
Arnaud Adant
31595e3496
Fix unix permissions
2022-07-19 14:54:31 -05:00
Kanthi Subramanian
3bd0c4fb42
Move flush timeout as a configuration variable. Change docker-compose to use environment variable for sink docker version.
2022-07-19 14:07:02 -04:00
Kanthi Subramanian
4ee8667b69
Added logic to flush records based on buffer size and number of records.
2022-07-15 16:23:19 -05:00
Kanthi Subramanian
601e900fab
Fixes to include decimal scale and precision when auto creating CH tables.
2022-07-14 14:46:35 -05:00
Kanthi Subramanian
5dfdf7beb4
Fixed bug with schema inference when auto creating tables, Write kafka offset data only when config variable is set to true.
2022-07-14 09:53:28 -05:00
Kanthi Subramanian
c436a14198
Added functionality to create auto tables for multiple primary key columns
2022-07-13 16:25:16 -05:00
Kanthi Subramanian
ba9f44e181
Enable GTID in MySQL docker image, added logic to set the version in ReplacingMergeTree to GTID if available, fallback to timestamp msecs(from debezium).
...
Fixed auto create tables for multiple primary keys.
2022-07-13 14:01:15 -05:00
Kanthi Subramanian
201ffc46bb
Refactoring, use a column to index map so that the Query template and the logic of adding to batch are preformed in the same order(JDBC bulk insert).
2022-07-12 11:18:56 -05:00
Kanthi Subramanian
fa660f6b4f
Fixed bugs in alter table and create table.
2022-07-10 14:55:15 -04:00
Kanthi Subramanian
e58474aacd
Changes to support alter table and create table.
2022-07-08 23:38:40 -04:00
Kanthi Subramanian
142f6bd76a
Changes to support alter table and create table.
2022-07-08 21:51:04 -04:00
Kanthi Subramanian
6c1f24524b
Added logic to extract primary key from Kafka connect key, used for alter table.
2022-07-07 22:18:26 -04:00
Kanthi Subramanian
28fc66213d
Added unit tests for data type mapper.
2022-07-06 08:13:08 -04:00
Kanthi Subramanian
d623cb5937
Added unit test directory to pom.xml
2022-07-03 19:22:14 -04:00
Kanthi Subramanian
994dc54113
Updated list of features.
2022-06-29 15:48:01 -04:00
Kanthi Subramanian
6f49271d78
More data types mapping for auto create tables.
2022-06-29 15:43:07 -04:00
Kanthi Subramanian
5f07dcdd28
Added more test cases for data type mapper.
2022-06-28 22:27:59 -04:00
Kanthi Subramanian
77daad80a6
Added more test cases for autocreate table, removed tableName passed to DbWriter.Fixed warnings
2022-06-27 12:18:01 -04:00
Kanthi Subramanian
d3a51d8ea5
Added test case for auto create table feature.
2022-06-27 11:23:32 -04:00
Kanthi Subramanian
7e4e63b208
Fixed imports.
2022-06-26 17:10:53 -04:00
Kanthi Subramanian
d02f7e8577
Added logic to create table in CH from record
2022-06-26 13:51:26 -04:00
Kanthi Subramanian
c3e01a8c81
Added logic to auto create tables - map kafka connect type to CH typesS
2022-06-24 16:44:54 -04:00
Kanthi Subramanian
831fb3332f
Added class to store data type mapping from kafka connect schema to Clickhouse schema, prep work for auto create tables.
2022-06-22 20:47:33 -04:00
Kanthi Subramanian
b055400be6
Added support for debezium microTimestampConverter.
2022-06-22 10:38:00 -04:00
Kanthi Subramanian
255a0e8bd3
Changed thread pool size to a configurable parameter, Changed the logic of removing records from the shared records queue.
2022-06-21 16:59:33 -04:00
Kanthi Subramanian
6db50db39c
Added documentation and sysbench read/write script.
2022-06-20 14:24:33 -04:00
Kanthi Subramanian
45dde57b8b
Added document for SysBench setup.
2022-06-20 11:52:55 -04:00
Kanthi Subramanian
3baa15301a
Added more debug logs, added synchronized to prevent race conditions
2022-06-19 17:36:03 -04:00
Kanthi Subramanian
df67e52d42
Refactoring KafkaOffsetWriter and added test case for MySQL JSON Data type.
2022-06-17 13:58:52 -04:00
Kanthi Subramanian
f4276ff82e
Updated README to include postgres as source database.
2022-06-16 17:29:04 -04:00
Kanthi Subramanian
7d1574a1ca
Added postgres docker compose setup, seed data for postgres and debezium connector configuration for PostgreSQL
2022-06-16 17:28:18 -04:00
Kanthi Subramanian
aa0366a954
Refactor DB connection to base class for both offset writer and payload writer logic.
2022-06-16 11:17:24 -04:00
Kanthi Subramanian
6cdd5a957e
Changed offset management from MV to AggregatingMergeTree
2022-06-15 21:36:19 -04:00
Kanthi Subramanian
4f31af4fe3
Updated README.md to include k8s pipeline setup.
2022-06-14 13:30:01 -04:00
Kanthi Subramanian
879d998517
Fixed logic of before/after struct. Added view as a separate table to avoid stack size exception(TOO_MUCH_RECURSION) in ClickHouse.
2022-06-14 12:55:39 -04:00
Kanthi Subramanian
de7f50e14f
Fixed NPE in ClickHouseConverter
2022-06-14 10:29:28 -04:00
Kanthi Subramanian
89da1a391d
Added more integration tests for DbKafkaOffsetWriter
2022-06-14 09:58:26 -04:00
Kanthi Subramanian
327b65c0a2
Updated README.md with new project title. Added tests for DbKafkaOffsetWriter
2022-06-13 14:35:34 -04:00
Kanthi Subramanian
831c69f50a
Added configuration parameter to enable/disable kafka offset management and table name.
2022-06-13 10:38:04 -04:00
Kanthi Subramanian
a359c7ae59
Added logic to retrieve the topic partition and offset for evey block. Added unit test.
2022-06-12 15:58:59 -04:00
Kanthi Subramanian
c6b79b6a1a
Added kafka lag dashboard
2022-06-10 21:41:09 -04:00
