Andrey, I mean the queue of update events that is collected on backup nodes and flushed to listening clients in case of topology change.
-Val On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev <andrewkor...@hotmail.com> wrote: > Val, > > Could you please elaborate what you mean by "updates queue" you plan to > maintain on the primary and backup nodes? > > Thanks > Andrey > > > Date: Fri, 24 Jul 2015 17:51:48 +0300 > > Subject: Re: Continuous queries changes > > From: yzhda...@apache.org > > To: dev@ignite.incubator.apache.org > > > > Val, > > > > I have idea on how to clean up backup queue. > > > > 1. Our communication uses acks. So, you can determine [on server node] > > whether client received the update from local server or not. I think you > > can easily change existing code to get notifications on ack receiving > (this > > way you dont need to introduce your own acks). > > 2. How do you know when evict from backup? Each message that client acks > > corresponds to some per-partition long value you talked above (great > idea, > > btw!). Servers can exchange per-partition long value that corresponds to > > the latest acked message and that's the way how backups can safely evict > > from the queue. > > > > Let me know if you have questions. > > > > --Yakov > > > > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko < > valentin.kuliche...@gmail.com > > >: > > > > > Igniters, > > > > > > Based on discussions with our users I came to conclusion that our > > > continuous query implementation is not good enough for use cases with > > > strong consistency requirements, because there is a possibility to lose > > > updates in case of topology change. > > > > > > So I started working on > https://issues.apache.org/jira/browse/IGNITE-426 > > > and I hope to finish in in couple of weeks so that we can include it in > > > next release. > > > > > > I have the following design in mind: > > > > > > - Maintain updates queue on backup node(s) in addition to primary > node. > > > - If primary node crushes, this queue is flushed to listening > clients. > > > - To avoid notification duplicates we will have a per-partition > update > > > counter. Once an entry in some partition is updated, counter for > this > > > partition is incremented on both primary and backups. The value of > this > > > counter is also sent along with the update to the client, which also > > > maintains the copy of this mapping. If at some moment it receives an > > > update > > > with the counter less than in its local map, this update is a > duplicate > > > and > > > can be discarded. > > > - Also need to figure out the best way to clean the backup queue if > > > topology is stable. Will be happy to hear any suggestions :) > > > > > > To make all this work we also need to implement thread-per-partition > mode > > > in atomic cache, because now updates order on backup nodes can differ > from > > > the primary node: https://issues.apache.org/jira/browse/IGNITE-104. > I'm > > > already working on this. > > > > > > Feel free to share your thoughts! > > > > > > -Val > > > > >
