Java與大數據架構：3. Sync Clickhouse with MySQL_MongoDB.pdf

《Java與大數據架構：3. Sync Clickhouse with MySQL_MongoDB.pdf》由會員分享，可在線閱讀，更多相關《Java與大數據架構：3. Sync Clickhouse with MySQL_MongoDB.pdf（38頁珍藏版）》請在三個皮匠報告上搜索。

1、Sync Clickhouse with MySQL/MongoDB Company: Xiaoxin Tech. Industry: Education Team: Big Data Leader: About 100 billion data this year till now 30 million users We use Clickhouse in our daily tasks Chanllenges Complex Datasource Chanllenges Frequent Updates Chanllenges Possible Solutions 1. Replay bi

2、nlog/oplog CRUD directly Cant update/delete table frequently in Clickhouse Possible Solutions 2. MySQL Engine Not suitable for big tables Not suitable for MongoDB Possible Solutions 3. Reinit whole table every day Possible Solutions 4. CollapsingMergeTree FINAL is slow GROUP BY id HAVING sum(sign)0

3、Need to use GROUP BY in every query Not suitable for multi-column primary key Our Solution: PTS Key Features Only one config file needed for a new Clickhouse table Init and keep syncing data in one app for a table Sync multiple data source to Clickhouse in minutes PTS Provider Transform Sinker Major Provider Must Listen Merge sharding tables SolidKey Typical Provider Config Type: mysql,/ mysql, m