Hi, I started to work on ignite-426 and see that current implementation of thread-per-partition approach required for this feature has serious impact on performance.
As I understand thread-per-partition is required to avoid issue with update order on backup nodes: now it is possible that on primary node updates were executed 'update 1', 'update 2' and on backup nodes update messages can be handled like <update 2>, <update 1>. But now backup will eventually have correct value due to the fact that cache entry version grows, so 'update 2' will always have version greater than 'update 1'. What if on backup we will sort continuous query events by cache update version? In this case when backup queue is flushed messages will be correctly ordered and thread-per-partition mode is not needed. Do you see any problems with this approach, does it worth to try this? Thanks On Thu, Jul 23, 2015 at 8:53 AM, Valentin Kulichenko < [email protected]> wrote: > Igniters, > > Based on discussions with our users I came to conclusion that our > continuous query implementation is not good enough for use cases with > strong consistency requirements, because there is a possibility to lose > updates in case of topology change. > > So I started working on https://issues.apache.org/jira/browse/IGNITE-426 > and I hope to finish in in couple of weeks so that we can include it in > next release. > > I have the following design in mind: > > - Maintain updates queue on backup node(s) in addition to primary node. > - If primary node crushes, this queue is flushed to listening clients. > - To avoid notification duplicates we will have a per-partition update > counter. Once an entry in some partition is updated, counter for this > partition is incremented on both primary and backups. The value of this > counter is also sent along with the update to the client, which also > maintains the copy of this mapping. If at some moment it receives an > update > with the counter less than in its local map, this update is a duplicate > and > can be discarded. > - Also need to figure out the best way to clean the backup queue if > topology is stable. Will be happy to hear any suggestions :) > > To make all this work we also need to implement thread-per-partition mode > in atomic cache, because now updates order on backup nodes can differ from > the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm > already working on this. > > Feel free to share your thoughts! > > -Val >
