Val, Could you please elaborate what you mean by "updates queue" you plan to maintain on the primary and backup nodes?
Thanks Andrey > Date: Fri, 24 Jul 2015 17:51:48 +0300 > Subject: Re: Continuous queries changes > From: yzhda...@apache.org > To: dev@ignite.incubator.apache.org > > Val, > > I have idea on how to clean up backup queue. > > 1. Our communication uses acks. So, you can determine [on server node] > whether client received the update from local server or not. I think you > can easily change existing code to get notifications on ack receiving (this > way you dont need to introduce your own acks). > 2. How do you know when evict from backup? Each message that client acks > corresponds to some per-partition long value you talked above (great idea, > btw!). Servers can exchange per-partition long value that corresponds to > the latest acked message and that's the way how backups can safely evict > from the queue. > > Let me know if you have questions. > > --Yakov > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko <valentin.kuliche...@gmail.com > >: > > > Igniters, > > > > Based on discussions with our users I came to conclusion that our > > continuous query implementation is not good enough for use cases with > > strong consistency requirements, because there is a possibility to lose > > updates in case of topology change. > > > > So I started working on https://issues.apache.org/jira/browse/IGNITE-426 > > and I hope to finish in in couple of weeks so that we can include it in > > next release. > > > > I have the following design in mind: > > > > - Maintain updates queue on backup node(s) in addition to primary node. > > - If primary node crushes, this queue is flushed to listening clients. > > - To avoid notification duplicates we will have a per-partition update > > counter. Once an entry in some partition is updated, counter for this > > partition is incremented on both primary and backups. The value of this > > counter is also sent along with the update to the client, which also > > maintains the copy of this mapping. If at some moment it receives an > > update > > with the counter less than in its local map, this update is a duplicate > > and > > can be discarded. > > - Also need to figure out the best way to clean the backup queue if > > topology is stable. Will be happy to hear any suggestions :) > > > > To make all this work we also need to implement thread-per-partition mode > > in atomic cache, because now updates order on backup nodes can differ from > > the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm > > already working on this. > > > > Feel free to share your thoughts! > > > > -Val > >
