Kanthi Subramanian
575e9a51b6
Added README and architecture proposal for CLI.
2023-06-07 10:55:06 -04:00
Kanthi Subramanian
d56d26a7fa
Fix document links.
2023-04-04 14:34:39 -04:00
Kanthi Subramanian
9915d22800
Merge lightweight connector
2023-04-04 13:29:46 -04:00
Kanthi
d0e3706725
Update mutable_data.md
2022-12-18 14:31:09 -05:00
Kanthi
bdc2d1f84a
Update mutable_data.md
2022-12-18 14:26:00 -05:00
Kanthi Subramanian
81414eb5a7
Updated Image for README.md
2022-12-07 14:44:50 -05:00
Kanthi Subramanian
0b9d5b511d
Added logic to store LSN for postgreSQL
2022-11-29 14:23:21 -05:00
Kanthi Subramanian
aaa63c6fe8
Updated postgres configuration for debezium 2.x and Confluent schema registry
2022-11-24 19:01:56 -05:00
Kanthi Subramanian
88ed8d38e8
Updated uptime and changed metrics from msecs to seconds.
2022-11-02 15:01:24 -04:00
Kanthi
da214285da
Merge branch 'develop' into grafana_dashboard_fixes
2022-10-23 14:06:45 -04:00
Kanthi Subramanian
15d9e9ba56
Added lag metrics in Grafana dashboard.
2022-10-23 13:41:21 -04:00
Kanthi Subramanian
1e24f412f1
Updated troubleshooting doc
2022-10-19 10:47:54 -04:00
Kanthi Subramanian
ce76f4e895
Resolved conflicts
2022-10-17 17:43:12 -04:00
Kanthi Subramanian
9564451357
Updating README.md
2022-10-17 14:22:59 -04:00
Kanthi
e3e3ce823c
Revert "77 add sakila employees database"
2022-09-19 11:52:21 -04:00
Kanthi Subramanian
fcf50b924e
Added documentation for Confluent Schema Registry REST API.
2022-09-19 11:50:15 -04:00
Kanthi Subramanian
a5dee23de1
Removed all references to clickhouse.table.name
2022-09-08 17:27:56 -04:00
Kanthi
fd3038e33a
Merge pull request #60 from Altinity/set_binlog_position_debezium
...
Manually set binlog position before starting streaming.
2022-08-29 08:26:01 -04:00
Kanthi Subramanian
7002bcd522
Update debezium setup instructions.
2022-08-29 17:34:42 +05:30
Kanthi Subramanian
3feb609454
Instructions to set the binlog position manually before starting ingestion.
2022-08-29 13:35:23 +05:30
Kanthi Subramanian
0a1567a4b4
Instructions to set the binlog position manually before starting ingestion.
2022-08-27 17:27:11 +05:30
Kanthi Subramanian
2e13fd4964
Added kubernetes pod for debezium.
2022-08-25 08:57:24 -04:00
Kanthi Subramanian
1a0cabffcb
Added redpanda UI(console), Strimzi debezium image, Added confluent schema registry ks8 manifest.
2022-08-24 02:09:44 -04:00
Kanthi Subramanian
f2d4be6277
Added redpanda 3 node cluster configuration and schema registry confluent.
2022-08-22 06:45:13 -04:00
Kanthi Subramanian
a4f605e886
Renamed docker compose file to follow the override format. Added configuration changes to connect to external MySQL Database.
2022-08-18 01:59:12 -04:00
Kanthi Subramanian
3a9f800c96
Cleanup docker compose files to remove duplicates and use the docker override syntax
2022-08-13 18:30:11 +05:30
Kanthi Subramanian
edef4968a8
Fixed logic of auto create table for empty primary key.
...
Load ontime airline dataset.
2022-08-11 09:54:06 -04:00
Kanthi Subramanian
08f9806916
Updated README instructions.
2022-08-02 17:46:37 -04:00
Kanthi Subramanian
0f851a1996
Fixed logic of retry for auto creating table
2022-08-02 17:41:38 -04:00
Kanthi Subramanian
9c6b376667
Added partition to offset metrics(Prometheus)
2022-07-27 11:01:52 -04:00
Kanthi Subramanian
ab42714f88
Added a map of topics to DBWriter instances to cover scenario where messages of multiple topics will be sent to the same Kafka SinkTask.
...
Fixed bug with extra addBatch
Added logic to check for indexes before inserting additional metadata
Added logic to not remove records if the addBatch was unsuccessful.
2022-07-19 17:41:52 -04:00
Kanthi Subramanian
3bd0c4fb42
Move flush timeout as a configuration variable. Change docker-compose to use environment variable for sink docker version.
2022-07-19 14:07:02 -04:00
Kanthi Subramanian
4ee8667b69
Added logic to flush records based on buffer size and number of records.
2022-07-15 16:23:19 -05:00
Kanthi Subramanian
c436a14198
Added functionality to create auto tables for multiple primary key columns
2022-07-13 16:25:16 -05:00
Kanthi Subramanian
ba9f44e181
Enable GTID in MySQL docker image, added logic to set the version in ReplacingMergeTree to GTID if available, fallback to timestamp msecs(from debezium).
...
Fixed auto create tables for multiple primary keys.
2022-07-13 14:01:15 -05:00
Kanthi Subramanian
831fb3332f
Added class to store data type mapping from kafka connect schema to Clickhouse schema, prep work for auto create tables.
2022-06-22 20:47:33 -04:00
Kanthi Subramanian
255a0e8bd3
Changed thread pool size to a configurable parameter, Changed the logic of removing records from the shared records queue.
2022-06-21 16:59:33 -04:00
Kanthi Subramanian
6db50db39c
Added documentation and sysbench read/write script.
2022-06-20 14:24:33 -04:00
Kanthi Subramanian
45dde57b8b
Added document for SysBench setup.
2022-06-20 11:52:55 -04:00
Kanthi Subramanian
c6b79b6a1a
Added kafka lag dashboard
2022-06-10 21:41:09 -04:00
Kanthi Subramanian
cc9742f538
Added README for explaining Updates/Deletes. Refactor logic of passing ReplacingMergeTree delete column.
2022-06-06 18:20:36 -04:00
Kanthi Subramanian
e1334e94ce
Added README for explaining Updates/Deletes.
2022-06-06 13:14:04 -04:00
Kanthi Subramanian
b36dbac88d
Updated pre-requisites for local setup
2022-05-31 13:29:39 -04:00
Kanthi Subramanian
690334f3a7
Changed the sign column to a Sink configuration variable.
2022-05-26 16:29:26 -04:00
Kanthi Subramanian
0d3a29dbc3
Updated Local setup instructions.
2022-05-24 09:50:03 -04:00
Vladislav Klimenko
dd01674b14
extract kubectl finalization
2022-05-12 17:20:16 +03:00
Vladislav Klimenko
6e76e34172
make port forward as a background process
2022-05-12 17:18:35 +03:00
Vladislav Klimenko
29f2454106
polish pipeline
2022-05-12 17:09:38 +03:00
Vladislav Klimenko
ebb0af59b9
adjust port forwarder
2022-05-12 17:06:20 +03:00
Vladislav Klimenko
7e2c18f134
extract port forward
2022-05-12 16:57:17 +03:00
