valitydev/clickhouse-sink-connector
14
0
Fork 0
mirror of https://github.com/valitydev/clickhouse-sink-connector.git synced 2024-11-07 02:55:23 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
192 Commits 8 Branches 0 Tags 202 MiB
fd67311834
Commit Graph

93 Commits

Author SHA1 Message Date
Kanthi Subramanian
4451438f71 Added logic to check if the column data type is DateTime64 for limiting the range. 2022-05-05 18:01:20 -04:00
Kanthi Subramanian
3d7fb07072 Added unit test case and logic to limit the DateTime values to clickhouse supported datetime ranges. 2022-05-05 12:13:42 -04:00
Kanthi Subramanian
e1cf390123 Added unit test case and logic to limit the Date values to clickhouse supported date ranges. 2022-05-05 11:09:05 -04:00
Vladislav Klimenko
fe0cdf1a8b unify connecto manifests and add some logging 2022-05-05 12:10:03 +03:00
Kanthi Subramanian
e67c9cd8e6 Fixed transformation of blob to string(hex). Changed prometheus reporter to use Counter instead of Gauge. 2022-05-04 11:21:11 -04:00
Kanthi Subramanian
d84198609b Added logic to use micrometer library and expose prometheus port for writing metrics. Added test case for deduplication. 2022-05-03 22:26:57 -04:00
Kanthi Subramanian
44642e953a Fixed bug with retrieving configuration of raw data column. 2022-05-02 23:05:26 -04:00
Kanthi Subramanian
c40c0f2d58 Fixed bug when topic2tablemap is not passed and added logic to update the map to store topic to table mapping. 2022-04-29 13:09:34 -04:00
Kanthi Subramanian
67a493c629 Updated documentation. 2022-04-28 17:20:19 -04:00
Kanthi Subramanian
d6531dfc9b Added variable to define the raw data column and enable/disable flag. 2022-04-28 14:11:42 -04:00
Kanthi Subramanian
7852969ccf Added function to convert Kafka Connect Struct to JSON and added test case. 2022-04-28 12:49:03 -04:00
Kanthi Subramanian
c350e95003 Refactoring - Move addKafkaMetadata to a separate class. 2022-04-28 10:15:02 -04:00
Kanthi Subramanian
b3c967ae91 Added logic to automatically map table name to topic name if not provided in the configuration. Added python test to query clickhouse using clickhouse python library. 2022-04-27 17:16:37 -04:00
Kanthi Subramanian
a2c9cf736b Modified data structure to store buffer records as HashMap of topicName to list of records. 2022-04-27 12:30:37 -04:00
Kanthi Subramanian
57ec2e84d8 Changed batch insert input function query syntax. Added logic to insert fake data for Products table to test multiple topics ingestion. 2022-04-26 22:17:48 -04:00
Kanthi Subramanian
8e7ded8f44 Added support for multiple topics using topics2tablemap. 2022-04-25 22:36:31 -04:00
Kanthi Subramanian
a1e1e3fe66 Change flush time to a kafka connect configuration variable. 2022-04-25 12:04:44 -04:00
Kanthi Subramanian
d52ff6ed2b Enable writing kafka metadata records. 2022-04-24 20:04:39 -04:00
Kanthi Subramanian
96a81778cd Added support for multiple tasks in Sink. 2022-04-24 20:04:15 -04:00
Kanthi Subramanian
cd9ca53d54 Added dropwizard metrics library to track memory usage and timers. 2022-04-20 12:27:53 -04:00
Kanthi Subramanian
0e109987f4 Refactoring, added unit tests for data types conversion. Added support for BLOB types. 2022-04-19 21:12:35 -04:00
Kanthi Subramanian
bfe952637a Merge remote-tracking branch 'origin/main' 
# Conflicts:
#	src/main/java/com/altinity/clickhouse/sink/connector/db/DbWriter.java
 2022-04-19 09:14:33 -04:00
Kanthi Subramanian
6382ccffe2 Changes to support scenario when kafka connect columns dont match Clickhouse columns. 2022-04-19 09:12:05 -04:00
Vladislav Klimenko
dc153138fe formatter 2022-04-19 15:40:31 +03:00
Vladislav Klimenko
10d02f99af keep deduplicator in the loop 2022-04-19 14:40:23 +03:00
Vladislav Klimenko
1c45bf4410 support possible off status of the deduplicator 2022-04-19 14:39:56 +03:00
Vladislav Klimenko
a0293ccf9f set OFF as default deduplicator policy 2022-04-19 14:39:28 +03:00
Vladislav Klimenko
97cf6ac08e clarify valiator 2022-04-19 14:39:02 +03:00
Vladislav Klimenko
939b2ab336 Introduce DeDuplication policy OFF 2022-04-19 14:38:25 +03:00
Kanthi Subramanian
a1ee03e601 Fixed writing of timestamp and key - kafka columns. 2022-04-16 09:11:37 -04:00
Kanthi Subramanian
e7f66e436e Optimized imports. 2022-04-14 14:45:00 -04:00
Kanthi Subramanian
8a5dee731a Added key to kafka metadata columns and updated clickhouse schema. Fixed warnings. 2022-04-14 14:37:39 -04:00
Kanthi Subramanian
542ce6e466 Added support for timestamp data type, updated SQL scripts. 2022-04-14 14:15:06 -04:00
Kanthi Subramanian
bcfe062926 Changed name of kafka metadata columns 2022-04-14 11:17:13 -04:00
Kanthi Subramanian
961af26d51 Added logic to make storing kafka metadata columns as a configurable parameter in Kafka connect sink configuration. 2022-04-13 10:19:16 -04:00
Kanthi Subramanian
899212a672 Merge remote-tracking branch 'origin/main' 2022-04-13 09:08:22 -04:00
Kanthi Subramanian
fc53044c08 Added support for Decimal Type. 2022-04-13 09:08:17 -04:00
Vladislav Klimenko
63cb8ea262 formatter 2022-04-13 15:28:30 +03:00
Vladislav Klimenko
f7b207996d extract dedup config 2022-04-13 15:28:21 +03:00
Vladislav Klimenko
9dd85df8aa clarify dedup 2022-04-13 15:18:25 +03:00
Vladislav Klimenko
41d6b35cce replace with new records 2022-04-13 15:07:53 +03:00
Vladislav Klimenko
314b64b39d config property 2022-04-13 15:07:28 +03:00
Vladislav Klimenko
f9ba2550e7 add DeDUp policy to config 2022-04-13 15:07:08 +03:00
Vladislav Klimenko
cf49d0c8a7 introduce DeDuplicationPolicyValidator 2022-04-13 15:06:45 +03:00
Vladislav Klimenko
143c70c4dc intorduce DeDuplicationPolicy 2022-04-13 13:59:58 +03:00
Vladislav Klimenko
487f345bb0 clarify dedup logic 2022-04-13 13:59:41 +03:00
Kanthi Subramanian
957b731f56 Added support for TIME and other Integer columns. 2022-04-12 18:27:35 -04:00
Kanthi Subramanian
53f53ea814 Fixed comparison of Kafka metadata column name. 2022-04-12 14:07:30 -04:00
Kanthi Subramanian
c70183f37d Store kafka metadata columns as Enum. 2022-04-12 13:28:29 -04:00
Kanthi Subramanian
d9fe27e6dc Fixed merge conflict 2022-04-12 09:57:15 -04:00