Folks, I thought more about this and came up with a WAL page formal that we can maintain in a separate queue and flush to persistent store. This allows to recover any lost queue data and also allows to rebuild a cluster from scratch.
Dmitriy, I understand that you were inclined to have redundancy of write behind store buffers for primary node on backup nodes as well and flush them in case of failure. Does the WAL page idea fit in your mind? Else we could go with pure redundancy based solution. Thoughts? On 6 Jul 2015 16:15, "Atri Sharma" <[email protected]> wrote: > Dmitriy, > > I created the ticket. Please let me know your thoughts around this. > > Once we have something in place, I will create page on wiki. > > On Mon, Jul 6, 2015 at 1:47 PM, Dmitriy Setrakyan <[email protected]> > wrote: > >> On Mon, Jul 6, 2015 at 1:13 AM, Atri Sharma <[email protected]> wrote: >> >> > Sounds good. >> > >> > I also propose controlling this behaviour through user settable option >> > (since this might potentially have some performance implications). So if >> > the user is doing ad hoc analytics, he/she might not care much about >> bit of >> > data loss and might prefer the performance gain instead. >> > >> >> I agree. Atri, can you please create a ticket describing this behavior? We >> can move this discussion there. >> >> >> > >> > On Mon, Jul 6, 2015 at 1:40 PM, Dmitriy Setrakyan < >> [email protected]> >> > wrote: >> > >> > > On Mon, Jul 6, 2015 at 1:00 AM, Atri Sharma <[email protected]> >> wrote: >> > > >> > > > Can some sort of persistent Write Ahead Logging help here? >> > > > >> > > >> > > I think so. Perhaps we can have the same type of queue on the backup >> > nodes >> > > which will only get flushed in case of primary node failure? >> > > >> > > >> > > > >> > > > On Mon, Jul 6, 2015 at 11:32 AM, Dmitriy Setrakyan < >> > > [email protected]> >> > > > wrote: >> > > > >> > > > > On Sun, Jul 5, 2015 at 10:58 PM, Vladimir Ozerov < >> > [email protected] >> > > > >> > > > > wrote: >> > > > > >> > > > > > Igniters, >> > > > > > >> > > > > > Can someone explain me how write-behind store should behave in >> case >> > > of >> > > > > node >> > > > > > stop when some changes has not been flushed to the underlying >> store >> > > > yet? >> > > > > > Are we guarantee that all pending changes will be flushed to the >> > > > > underlying >> > > > > > store? >> > > > > > >> > > > > > >> > > > > I think we should make the best effort to persist the un-flushed >> > > updates >> > > > in >> > > > > case of graceful exit. Of course, not guarantees can be provided, >> and >> > > if >> > > > a >> > > > > server stop abruptly, then some updates might be lost. >> > > > > >> > > > > >> > > > > > Vladimir. >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > Regards, >> > > > >> > > > Atri >> > > > *l'apprenant* >> > > > >> > > >> > >> > >> > >> > -- >> > Regards, >> > >> > Atri >> > *l'apprenant* >> > >> > > > > -- > Regards, > > Atri > *l'apprenant* >
