Re: Ignite-2.0 - Make near readers update async by default
Filed a ticket https://issues.apache.org/jira/browse/IGNITE-4665 --Yakov 2017-02-07 0:49 GMT+07:00 Dmitriy Setrakyan: > On Mon, Feb 6, 2017 at 12:29 AM, Yakov Zhdanov > wrote: > > > What users do not expect is that partitioned cache suddenly turns to > > replicated with replicas on client machines running in virtual > environment > > on higher latency network. I am a bit surprised nobody sees it. I predict > > we will soon start getting questions from our largest deployments =) > > > > Yakov, I tend to agree with you, even though up until now we have not > received any complaints in this regard. Let's add a new mode and call it, > say, NEAR_ASYNC or DHT_SYNC. When this mode is turned on, the near caches > will be updated asynchronously. I am assuming that we will take care of the > back pressure as we always do. > > Can you please create a ticket? > > > > > > --Yakov > > > > 2017-02-06 15:20 GMT+07:00 Vladimir Ozerov : > > > > > Yakov, > > > > > > I think forcing reads in TX or from primary is not what users expect by > > > default. So i would have this mode disabled by default. > > > > > > On Mon, Feb 6, 2017 at 10:40 AM, Yakov Zhdanov > > > wrote: > > > > > > > >It still seems that outdated reads will be *more* outdated with > async > > > > mode. > > > > >Also, is there a guarantee that a near-cache update will happen at > > all, > > > if > > > > >you introduce the async mode? > > > > > > > > We have the same guarantees for continuous queries - updates are sent > > to > > > > listener and no ack is required on grid level protocol (communication > > > > guaranteed delivery is used). If near node receives messages and > > process > > > > them, then the update should happen, if it does not receive messages > it > > > > should be thrown out of topology as message queue to it grows (slow > > > client > > > > limit) > > > > > > > > I do not want to operate "more outdated" or "less outdated" > > definitions. > > > To > > > > me both of them are pretty much the same :) Want up to date reads - > > read > > > > from primary or in TX, all other options may be "outdated". > > > > > > > > > Is this going to be the default flag? > > > > > > > > Well, I don't want to take decision at the moment, but having > DHT_SYNC > > > > seems very good option to me. PRIMARY_SYNC may stay default. All I > want > > > is > > > > to have opportunity to update near readers in async way. > > > > > > > > >Are you really suggesting that TX is committed without a guarantee > > that > > > > >near cache update happened? > > > > > > > > Do not see any issue here. You can ensure consistency and reread the > > > value > > > > in TX. Or you can enforce this by choosing FULL_SYNC for this update. > > > > > > > > Btw, is there any way to override configured writeSync mode? Seems > > pretty > > > > nice to have IgniteCache.withWriteSynchronizationMode(Mode m) > > > > > > > > >This would be a great optimization. It sounds like it could be done > > > > >independently from sync or async updates of near caches, no? > > > > > > > > I think so > > > > > > > > >I still don't see it. It is still possible for a near node to be > > alive, > > > > but > > > > >unresponsive. In this case, there is a possibility that a near cache > > > will > > > > >never be updated, even though the transaction has already been > > > committed. > > > > >This just does not seem kosher to me. > > > > > > > > See above example for continuous queries and slow client queue limit. > > > > > > > > Thanks! > > > > > > > > > > > > > > > > --Yakov > > > > > > > > 2017-02-04 12:12 GMT+07:00 Dmitriy Setrakyan >: > > > > > > > > > Hm... interesting. My questions are inline. > > > > > > > > > > On Mon, Jan 30, 2017 at 3:29 PM, Yakov Zhdanov < > yzhda...@apache.org> > > > > > wrote: > > > > > > > > > > > Guys, > > > > > > > > > > > > Currently when we do cache updates in FULL_SYNC mode we update > near > > > > > readers > > > > > > (near cache entries) synchronously. This is quite big drawback in > > > > > design, I > > > > > > think. I get each near reader update at cost of 1 extra backup > > > update. > > > > I > > > > > > think everyone understands that partitioned cache easily turns to > > > > > > replicated once near readers number increases. In TX cache cost > of > > > such > > > > > > updates doubles. > > > > > > > > > > > > I do not see any benefit on updating near entries in sync way. > > > Outdated > > > > > > reads can still be possible if I don't read from primary or in TX > > > > > context. > > > > > > > > > > > > > > > > It still seems that outdated reads will be *more* outdated with > async > > > > mode. > > > > > Also, is there a guarantee that a near-cache update will happen at > > all, > > > > if > > > > > you introduce the async mode? > > > > > > > > > > > > > > > > > So, what I suggest: > > > > > > 1. introduce flag for cahce - withSyncNearUpdates() or
Re: Ignite-2.0 - Make near readers update async by default
On Mon, Feb 6, 2017 at 12:29 AM, Yakov Zhdanovwrote: > What users do not expect is that partitioned cache suddenly turns to > replicated with replicas on client machines running in virtual environment > on higher latency network. I am a bit surprised nobody sees it. I predict > we will soon start getting questions from our largest deployments =) > Yakov, I tend to agree with you, even though up until now we have not received any complaints in this regard. Let's add a new mode and call it, say, NEAR_ASYNC or DHT_SYNC. When this mode is turned on, the near caches will be updated asynchronously. I am assuming that we will take care of the back pressure as we always do. Can you please create a ticket? > > --Yakov > > 2017-02-06 15:20 GMT+07:00 Vladimir Ozerov : > > > Yakov, > > > > I think forcing reads in TX or from primary is not what users expect by > > default. So i would have this mode disabled by default. > > > > On Mon, Feb 6, 2017 at 10:40 AM, Yakov Zhdanov > > wrote: > > > > > >It still seems that outdated reads will be *more* outdated with async > > > mode. > > > >Also, is there a guarantee that a near-cache update will happen at > all, > > if > > > >you introduce the async mode? > > > > > > We have the same guarantees for continuous queries - updates are sent > to > > > listener and no ack is required on grid level protocol (communication > > > guaranteed delivery is used). If near node receives messages and > process > > > them, then the update should happen, if it does not receive messages it > > > should be thrown out of topology as message queue to it grows (slow > > client > > > limit) > > > > > > I do not want to operate "more outdated" or "less outdated" > definitions. > > To > > > me both of them are pretty much the same :) Want up to date reads - > read > > > from primary or in TX, all other options may be "outdated". > > > > > > > Is this going to be the default flag? > > > > > > Well, I don't want to take decision at the moment, but having DHT_SYNC > > > seems very good option to me. PRIMARY_SYNC may stay default. All I want > > is > > > to have opportunity to update near readers in async way. > > > > > > >Are you really suggesting that TX is committed without a guarantee > that > > > >near cache update happened? > > > > > > Do not see any issue here. You can ensure consistency and reread the > > value > > > in TX. Or you can enforce this by choosing FULL_SYNC for this update. > > > > > > Btw, is there any way to override configured writeSync mode? Seems > pretty > > > nice to have IgniteCache.withWriteSynchronizationMode(Mode m) > > > > > > >This would be a great optimization. It sounds like it could be done > > > >independently from sync or async updates of near caches, no? > > > > > > I think so > > > > > > >I still don't see it. It is still possible for a near node to be > alive, > > > but > > > >unresponsive. In this case, there is a possibility that a near cache > > will > > > >never be updated, even though the transaction has already been > > committed. > > > >This just does not seem kosher to me. > > > > > > See above example for continuous queries and slow client queue limit. > > > > > > Thanks! > > > > > > > > > > > > --Yakov > > > > > > 2017-02-04 12:12 GMT+07:00 Dmitriy Setrakyan : > > > > > > > Hm... interesting. My questions are inline. > > > > > > > > On Mon, Jan 30, 2017 at 3:29 PM, Yakov Zhdanov > > > > wrote: > > > > > > > > > Guys, > > > > > > > > > > Currently when we do cache updates in FULL_SYNC mode we update near > > > > readers > > > > > (near cache entries) synchronously. This is quite big drawback in > > > > design, I > > > > > think. I get each near reader update at cost of 1 extra backup > > update. > > > I > > > > > think everyone understands that partitioned cache easily turns to > > > > > replicated once near readers number increases. In TX cache cost of > > such > > > > > updates doubles. > > > > > > > > > > I do not see any benefit on updating near entries in sync way. > > Outdated > > > > > reads can still be possible if I don't read from primary or in TX > > > > context. > > > > > > > > > > > > > It still seems that outdated reads will be *more* outdated with async > > > mode. > > > > Also, is there a guarantee that a near-cache update will happen at > all, > > > if > > > > you introduce the async mode? > > > > > > > > > > > > > > So, what I suggest: > > > > > 1. introduce flag for cahce - withSyncNearUpdates() or extra > > > > > CacheWriteSynchronizationMode.DHT_SYNC and make it default mode. > > > > > > > > > > > > > Is this going to be the default flag? > > > > > > > > > > > > > 2. Near entries are updated in async way > > > > > 2.1 in atomic mode together with backup updates > > > > > 2.2 in TX mode after tx is committed on primary > > > > > > > > > > > > Are you really suggesting that TX is committed without a guarantee > that > > >
Re: Ignite-2.0 - Make near readers update async by default
>It still seems that outdated reads will be *more* outdated with async mode. >Also, is there a guarantee that a near-cache update will happen at all, if >you introduce the async mode? We have the same guarantees for continuous queries - updates are sent to listener and no ack is required on grid level protocol (communication guaranteed delivery is used). If near node receives messages and process them, then the update should happen, if it does not receive messages it should be thrown out of topology as message queue to it grows (slow client limit) I do not want to operate "more outdated" or "less outdated" definitions. To me both of them are pretty much the same :) Want up to date reads - read from primary or in TX, all other options may be "outdated". > Is this going to be the default flag? Well, I don't want to take decision at the moment, but having DHT_SYNC seems very good option to me. PRIMARY_SYNC may stay default. All I want is to have opportunity to update near readers in async way. >Are you really suggesting that TX is committed without a guarantee that >near cache update happened? Do not see any issue here. You can ensure consistency and reread the value in TX. Or you can enforce this by choosing FULL_SYNC for this update. Btw, is there any way to override configured writeSync mode? Seems pretty nice to have IgniteCache.withWriteSynchronizationMode(Mode m) >This would be a great optimization. It sounds like it could be done >independently from sync or async updates of near caches, no? I think so >I still don't see it. It is still possible for a near node to be alive, but >unresponsive. In this case, there is a possibility that a near cache will >never be updated, even though the transaction has already been committed. >This just does not seem kosher to me. See above example for continuous queries and slow client queue limit. Thanks! --Yakov 2017-02-04 12:12 GMT+07:00 Dmitriy Setrakyan: > Hm... interesting. My questions are inline. > > On Mon, Jan 30, 2017 at 3:29 PM, Yakov Zhdanov > wrote: > > > Guys, > > > > Currently when we do cache updates in FULL_SYNC mode we update near > readers > > (near cache entries) synchronously. This is quite big drawback in > design, I > > think. I get each near reader update at cost of 1 extra backup update. I > > think everyone understands that partitioned cache easily turns to > > replicated once near readers number increases. In TX cache cost of such > > updates doubles. > > > > I do not see any benefit on updating near entries in sync way. Outdated > > reads can still be possible if I don't read from primary or in TX > context. > > > > It still seems that outdated reads will be *more* outdated with async mode. > Also, is there a guarantee that a near-cache update will happen at all, if > you introduce the async mode? > > > > > So, what I suggest: > > 1. introduce flag for cahce - withSyncNearUpdates() or extra > > CacheWriteSynchronizationMode.DHT_SYNC and make it default mode. > > > > Is this going to be the default flag? > > > > 2. Near entries are updated in async way > > 2.1 in atomic mode together with backup updates > > 2.2 in TX mode after tx is committed on primary > > > Are you really suggesting that TX is committed without a guarantee that > near cache update happened? > > I would also suggest to exclude near readers from lock acquisition/release > > steps. Only force updates. Updates order will be ensured by single > primary > > node and > > per-partition striping. > > > > This would be a great optimization. It sounds like it could be done > independently from sync or async updates of near caches, no? > > > > 3. Near readers do not respond to primary. Once primary fails near > readers > > get invalidated, if primary is alive then communication recovery ensures > > that message will be delivered to near. > > > > I still don't see it. It is still possible for a near node to be alive, but > unresponsive. In this case, there is a possibility that a near cache will > never be updated, even though the transaction has already been committed. > This just does not seem kosher to me. > > > > > > Please share your thoughts. > > > > --Yakov > > >
Re: Ignite-2.0 - Make near readers update async by default
Hm... interesting. My questions are inline. On Mon, Jan 30, 2017 at 3:29 PM, Yakov Zhdanovwrote: > Guys, > > Currently when we do cache updates in FULL_SYNC mode we update near readers > (near cache entries) synchronously. This is quite big drawback in design, I > think. I get each near reader update at cost of 1 extra backup update. I > think everyone understands that partitioned cache easily turns to > replicated once near readers number increases. In TX cache cost of such > updates doubles. > > I do not see any benefit on updating near entries in sync way. Outdated > reads can still be possible if I don't read from primary or in TX context. > It still seems that outdated reads will be *more* outdated with async mode. Also, is there a guarantee that a near-cache update will happen at all, if you introduce the async mode? > > So, what I suggest: > 1. introduce flag for cahce - withSyncNearUpdates() or extra > CacheWriteSynchronizationMode.DHT_SYNC and make it default mode. > Is this going to be the default flag? > 2. Near entries are updated in async way > 2.1 in atomic mode together with backup updates > 2.2 in TX mode after tx is committed on primary Are you really suggesting that TX is committed without a guarantee that near cache update happened? I would also suggest to exclude near readers from lock acquisition/release > steps. Only force updates. Updates order will be ensured by single primary > node and > per-partition striping. > This would be a great optimization. It sounds like it could be done independently from sync or async updates of near caches, no? > 3. Near readers do not respond to primary. Once primary fails near readers > get invalidated, if primary is alive then communication recovery ensures > that message will be delivered to near. > I still don't see it. It is still possible for a near node to be alive, but unresponsive. In this case, there is a possibility that a near cache will never be updated, even though the transaction has already been committed. This just does not seem kosher to me. > > Please share your thoughts. > > --Yakov >
Ignite-2.0 - Make near readers update async by default
Guys, Currently when we do cache updates in FULL_SYNC mode we update near readers (near cache entries) synchronously. This is quite big drawback in design, I think. I get each near reader update at cost of 1 extra backup update. I think everyone understands that partitioned cache easily turns to replicated once near readers number increases. In TX cache cost of such updates doubles. I do not see any benefit on updating near entries in sync way. Outdated reads can still be possible if I don't read from primary or in TX context. So, what I suggest: 1. introduce flag for cahce - withSyncNearUpdates() or extra CacheWriteSynchronizationMode.DHT_SYNC and make it default mode. 2. Near entries are updated in async way 2.1 in atomic mode together with backup updates 2.2 in TX mode after tx is committed on primary. I would also suggest to exclude near readers from lock acquisition/release steps. Only force updates. Updates order will be ensured by single primary node and per-partition striping. 3. Near readers do not respond to primary. Once primary fails near readers get invalidated, if primary is alive then communication recovery ensures that message will be delivered to near. Please share your thoughts. --Yakov