Guys, I added preliminary version of API for continuous queries to this ticket: https://issues.apache.org/jira/browse/IGNITE-1191
I would like to ask everyone interested in this topic to look at it and provide comments. -Val On Thu, Aug 13, 2015 at 4:48 AM, Semyon Boikov <[email protected]> wrote: > After more thinking about ordering on backups instead of > thread-per-partition I believe this approach won't work. > > First it is possible that primary executes for some key <update 1>, <update > 2> (from different threads), then crashes and only <update 1> message is > sent to backup, so backup won't be able to restore correct ordering. > > Also if the same partition can be updated concurrently it is not possible > to associate per-partition update counter with each update so it is not > clear how to deal with duplicated messages and backup acknowledgement. > > But since there is no thread-per-partition mode for transactional cache do > these issues affect transactional cache as well? > > > On Wed, Aug 12, 2015 at 4:49 PM, Semyon Boikov <[email protected]> > wrote: > > > Hi, > > > > I started to work on ignite-426 and see that current implementation of > > thread-per-partition approach required for this feature has serious > impact > > on performance. > > > > As I understand thread-per-partition is required to avoid issue with > > update order on backup nodes: now it is possible that on primary node > > updates were executed 'update 1', 'update 2' and on backup nodes update > > messages can be handled like <update 2>, <update 1>. > > > > But now backup will eventually have correct value due to the fact that > > cache entry version grows, so 'update 2' will always have version greater > > than 'update 1'. > > > > What if on backup we will sort continuous query events by cache update > > version? In this case when backup queue is flushed messages will be > > correctly ordered and thread-per-partition mode is not needed. > > > > Do you see any problems with this approach, does it worth to try this? > > > > Thanks > > > > > > On Thu, Jul 23, 2015 at 8:53 AM, Valentin Kulichenko < > > [email protected]> wrote: > > > >> Igniters, > >> > >> Based on discussions with our users I came to conclusion that our > >> continuous query implementation is not good enough for use cases with > >> strong consistency requirements, because there is a possibility to lose > >> updates in case of topology change. > >> > >> So I started working on > https://issues.apache.org/jira/browse/IGNITE-426 > >> and I hope to finish in in couple of weeks so that we can include it in > >> next release. > >> > >> I have the following design in mind: > >> > >> - Maintain updates queue on backup node(s) in addition to primary > node. > >> - If primary node crushes, this queue is flushed to listening > clients. > >> - To avoid notification duplicates we will have a per-partition > update > >> counter. Once an entry in some partition is updated, counter for this > >> partition is incremented on both primary and backups. The value of > this > >> counter is also sent along with the update to the client, which also > >> maintains the copy of this mapping. If at some moment it receives an > >> update > >> with the counter less than in its local map, this update is a > >> duplicate and > >> can be discarded. > >> - Also need to figure out the best way to clean the backup queue if > >> topology is stable. Will be happy to hear any suggestions :) > >> > >> To make all this work we also need to implement thread-per-partition > mode > >> in atomic cache, because now updates order on backup nodes can differ > from > >> the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm > >> already working on this. > >> > >> Feel free to share your thoughts! > >> > >> -Val > >> > > > > >
