[
https://issues.apache.org/jira/browse/IGNITE-17700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov updated IGNITE-17700:
-------------------------------------
Labels: IEP-59 ise (was: IEP-59)
> Online CDC
> ----------
>
> Key: IGNITE-17700
> URL: https://issues.apache.org/jira/browse/IGNITE-17700
> Project: Ignite
> Issue Type: Improvement
> Reporter: Nikolay Izhikov
> Priority: Major
> Labels: IEP-59, ise
>
> Right now, timeout between CDC event happens and consumer notified about it
> is relatively big.
> It sums from the following:
> * {{DataStorageConfiguration#walForceArchiveTimeout}}.
> * segment archivation time.
> * {{CdcConfiguration#checkFrequence}}
> * time to consume previous events.
> Advantages of this way described in
> [IEP-59|https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+Change+Data+Capture]
> So if user wants to minimize this timeouts we can implement the following:
> * Collects cdc event inside fixed-size buffer in node.
> * Consume events from buffer by CDCConsumer inside node process.
> * Store WALPointer of last event added to buffer.
> * On buffer overflow stop push events to it.
> ** Fallback to WAL segments iteration
> ** When iteration catch up WAL tail switch back to the buffer.
> * On node failover recover consumer with the cdc state as in ignite-cdc.
> With this approach most of the time CDCConsumer will consume events in near
> realtime fashion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
