Hi Nikolay, The idea is good. But what do you think to integrate these ideas into WAL-G project? https://github.com/wal-g/wal-g It's a well-known tool that is already used to stream WAL for PostgreSQL, MySQL, and MongoDB. The advantages are integration with S3, GCP, Azure out of the box, encryption, and compression.
ср, 14 окт. 2020 г. в 14:21, Nikolay Izhikov <[email protected]>: > Hello, Igniters. > > I want to start a discussion of the new feature [1] > > CDC - capture data change. The feature allows the consumer to receive > online notifications about data record changes. > > It can be used in the following scenarios: > * Export data into some warehouse, full-text search, or > distributed log system. > * Online statistics and analytics. > * Wait and respond to some specific events or data changes. > > Propose to implement new IgniteCDC application as follows: > * Run on the server node host. > * Watches for the appearance of the WAL archive segments. > * Iterates it using existing WALIterator and notifies consumer of > each record from the segment. > > IgniteCDC features: > * Independence from the server node process (JVM) - issues and > failures of the consumer will not lead to server node instability. > * Notification guarantees and failover - i.e. CDC track and save > the pointer to the last consumed record. Continue notification from this > pointer in case of restart. > * Resilience for the consumer - it's not an issue when a consumer > temporarily consumes slower than data appear. > > WDYT? > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+CDC+-+Capture+Data+Change
