Kanthi Subramanian
|
142f6bd76a
|
Changes to support alter table and create table.
|
2022-07-08 21:51:04 -04:00 |
|
Kanthi Subramanian
|
6c1f24524b
|
Added logic to extract primary key from Kafka connect key, used for alter table.
|
2022-07-07 22:18:26 -04:00 |
|
Kanthi Subramanian
|
28fc66213d
|
Added unit tests for data type mapper.
|
2022-07-06 08:13:08 -04:00 |
|
Kanthi Subramanian
|
d623cb5937
|
Added unit test directory to pom.xml
|
2022-07-03 19:22:14 -04:00 |
|
Kanthi Subramanian
|
994dc54113
|
Updated list of features.
|
2022-06-29 15:48:01 -04:00 |
|
Kanthi Subramanian
|
6f49271d78
|
More data types mapping for auto create tables.
|
2022-06-29 15:43:07 -04:00 |
|
Kanthi Subramanian
|
5f07dcdd28
|
Added more test cases for data type mapper.
|
2022-06-28 22:27:59 -04:00 |
|
Kanthi Subramanian
|
77daad80a6
|
Added more test cases for autocreate table, removed tableName passed to DbWriter.Fixed warnings
|
2022-06-27 12:18:01 -04:00 |
|
Kanthi Subramanian
|
d3a51d8ea5
|
Added test case for auto create table feature.
|
2022-06-27 11:23:32 -04:00 |
|
Kanthi Subramanian
|
7e4e63b208
|
Fixed imports.
|
2022-06-26 17:10:53 -04:00 |
|
Kanthi Subramanian
|
d02f7e8577
|
Added logic to create table in CH from record
|
2022-06-26 13:51:26 -04:00 |
|
Kanthi Subramanian
|
c3e01a8c81
|
Added logic to auto create tables - map kafka connect type to CH typesS
|
2022-06-24 16:44:54 -04:00 |
|
Kanthi Subramanian
|
831fb3332f
|
Added class to store data type mapping from kafka connect schema to Clickhouse schema, prep work for auto create tables.
|
2022-06-22 20:47:33 -04:00 |
|
Kanthi Subramanian
|
b055400be6
|
Added support for debezium microTimestampConverter.
|
2022-06-22 10:38:00 -04:00 |
|
Kanthi Subramanian
|
255a0e8bd3
|
Changed thread pool size to a configurable parameter, Changed the logic of removing records from the shared records queue.
|
2022-06-21 16:59:33 -04:00 |
|
Kanthi Subramanian
|
6db50db39c
|
Added documentation and sysbench read/write script.
|
2022-06-20 14:24:33 -04:00 |
|
Kanthi Subramanian
|
45dde57b8b
|
Added document for SysBench setup.
|
2022-06-20 11:52:55 -04:00 |
|
Kanthi Subramanian
|
3baa15301a
|
Added more debug logs, added synchronized to prevent race conditions
|
2022-06-19 17:36:03 -04:00 |
|
Kanthi Subramanian
|
df67e52d42
|
Refactoring KafkaOffsetWriter and added test case for MySQL JSON Data type.
|
2022-06-17 13:58:52 -04:00 |
|
Kanthi Subramanian
|
f4276ff82e
|
Updated README to include postgres as source database.
|
2022-06-16 17:29:04 -04:00 |
|
Kanthi Subramanian
|
7d1574a1ca
|
Added postgres docker compose setup, seed data for postgres and debezium connector configuration for PostgreSQL
|
2022-06-16 17:28:18 -04:00 |
|
Kanthi Subramanian
|
aa0366a954
|
Refactor DB connection to base class for both offset writer and payload writer logic.
|
2022-06-16 11:17:24 -04:00 |
|
Kanthi Subramanian
|
6cdd5a957e
|
Changed offset management from MV to AggregatingMergeTree
|
2022-06-15 21:36:19 -04:00 |
|
Kanthi Subramanian
|
4f31af4fe3
|
Updated README.md to include k8s pipeline setup.
|
2022-06-14 13:30:01 -04:00 |
|
Kanthi Subramanian
|
879d998517
|
Fixed logic of before/after struct. Added view as a separate table to avoid stack size exception(TOO_MUCH_RECURSION) in ClickHouse.
|
2022-06-14 12:55:39 -04:00 |
|
Kanthi Subramanian
|
de7f50e14f
|
Fixed NPE in ClickHouseConverter
|
2022-06-14 10:29:28 -04:00 |
|
Kanthi Subramanian
|
89da1a391d
|
Added more integration tests for DbKafkaOffsetWriter
|
2022-06-14 09:58:26 -04:00 |
|
Kanthi Subramanian
|
327b65c0a2
|
Updated README.md with new project title. Added tests for DbKafkaOffsetWriter
|
2022-06-13 14:35:34 -04:00 |
|
Kanthi Subramanian
|
831c69f50a
|
Added configuration parameter to enable/disable kafka offset management and table name.
|
2022-06-13 10:38:04 -04:00 |
|
Kanthi Subramanian
|
a359c7ae59
|
Added logic to retrieve the topic partition and offset for evey block. Added unit test.
|
2022-06-12 15:58:59 -04:00 |
|
Kanthi Subramanian
|
c6b79b6a1a
|
Added kafka lag dashboard
|
2022-06-10 21:41:09 -04:00 |
|
Kanthi Subramanian
|
0ea6dc80ea
|
Added script to create debezium connector for sysbench tables.
|
2022-06-09 16:04:14 -04:00 |
|
Kanthi Subramanian
|
923a95ab6a
|
Updated mysql configuration to set max_connections to 5000. Added sysbench script to run load test of inserts.
|
2022-06-09 13:20:38 -04:00 |
|
Kanthi Subramanian
|
1885cc42c4
|
Updated README.md
|
2022-06-09 09:53:48 -04:00 |
|
Kanthi Subramanian
|
c5c683b8f6
|
Refactored construct of ClickhouseStruct so that state is not maintained in the functions. Added test case for DBWriter.
|
2022-06-08 16:35:01 -04:00 |
|
Kanthi Subramanian
|
6ec8e576a8
|
Fixed bug with persisting sign column.
|
2022-06-08 09:26:22 -04:00 |
|
Kanthi Subramanian
|
cc9742f538
|
Added README for explaining Updates/Deletes. Refactor logic of passing ReplacingMergeTree delete column.
|
2022-06-06 18:20:36 -04:00 |
|
Kanthi Subramanian
|
e1334e94ce
|
Added README for explaining Updates/Deletes.
|
2022-06-06 13:14:04 -04:00 |
|
Kanthi Subramanian
|
0451ddf684
|
Removed sign/version column from configuration. Added test case to validate ReplacingMergeTree and CollapsingMergeTree.
|
2022-06-06 12:18:38 -04:00 |
|
Kanthi Subramanian
|
95c2d75299
|
Refactoring to get sign/version column along with Table Engine during initialization of DBWriter
|
2022-06-06 10:08:18 -04:00 |
|
Kanthi Subramanian
|
78634b19fa
|
Added functions to extract version and sign columns from engine column in system tables.
|
2022-06-03 13:15:09 -04:00 |
|
Kanthi Subramanian
|
2a50183b28
|
Updated docker image to dockerhub for sink connector
|
2022-06-03 10:27:43 -04:00 |
|
Kanthi Subramanian
|
900c2e72f3
|
Added logic to read version column and to set it to timestamp(epoch in ms) for updates.
|
2022-06-01 17:18:44 -04:00 |
|
Kanthi Subramanian
|
d174c68f3e
|
Set flag in debezium connector to fix public key retrieval error.
|
2022-06-01 12:52:52 -04:00 |
|
Kanthi Subramanian
|
98fad54edc
|
Fixed bug with adding batch for insert statements.
|
2022-06-01 10:17:55 -04:00 |
|
Kanthi Subramanian
|
eebddf9b5f
|
Fixed quotes in get table engine from system table.
|
2022-05-31 18:11:11 -04:00 |
|
Kanthi Subramanian
|
a8cc587553
|
Fixed logic of setting sign column for updates.
|
2022-05-31 16:00:49 -04:00 |
|
Kanthi Subramanian
|
b36dbac88d
|
Updated pre-requisites for local setup
|
2022-05-31 13:29:39 -04:00 |
|
Kanthi Subramanian
|
c39844cd65
|
Refactoring to support retrieving before/after columns and logic to insert records(before and after separately) when its an update.
|
2022-05-31 11:18:14 -04:00 |
|
Kanthi Subramanian
|
456fbb0c8a
|
Added function to identify Clickhouse table engine using system tables.
|
2022-05-28 10:37:53 -04:00 |
|