[infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?
Hi, I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why "org.infinispan.distribution.RehashInProgressException: Timed out waiting for the transaction lock" is thrown? This is thrown DistTxInterceptor which is added by the InterceptorChainFactory: if (configuration.getCacheMode().isDistributed()) interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class)); else interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class)); However, this is a Hot Rod test and the cache is not configured with a transaction manager, so is DistTxInterceptor really needed? In fact, why a TxInterceptor if no transaction manager is configured? (this kinda goes back to Mircea's email earlier today about a transactions enabled/disabled flag) Cheers, -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?
On 13 Sep 2011, at 17:22, Galder Zamarreño wrote: > Hi, > > I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why > "org.infinispan.distribution.RehashInProgressException: Timed out waiting for > the transaction lock" is thrown? > > This is thrown DistTxInterceptor which is added by the > InterceptorChainFactory: > > if (configuration.getCacheMode().isDistributed()) > > interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class)); > else > > interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class)); > > However, this is a Hot Rod test and the cache is not configured with a > transaction manager, so is DistTxInterceptor really needed? > > In fact, why a TxInterceptor if no transaction manager is configured? (this > kinda goes back to Mircea's email earlier today about a transactions > enabled/disabled flag) All valid concerns, IMO. 5.1 will clarify this by not adding Tx interceptors for non tx caches. > > Cheers, > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?
On Sep 13, 2011, at 6:38 PM, Mircea Markus wrote: > > On 13 Sep 2011, at 17:22, Galder Zamarreño wrote: > >> Hi, >> >> I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why >> "org.infinispan.distribution.RehashInProgressException: Timed out waiting >> for the transaction lock" is thrown? >> >> This is thrown DistTxInterceptor which is added by the >> InterceptorChainFactory: >> >> if (configuration.getCacheMode().isDistributed()) >> >> interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class)); >> else >> >> interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class)); >> >> However, this is a Hot Rod test and the cache is not configured with a >> transaction manager, so is DistTxInterceptor really needed? >> >> In fact, why a TxInterceptor if no transaction manager is configured? (this >> kinda goes back to Mircea's email earlier today about a transactions >> enabled/disabled flag) > All valid concerns, IMO. > 5.1 will clarify this by not adding Tx interceptors for non tx caches. Is this captured in some other JIRA? I'd like to reference the test and the JIRA from ISPN-1123 (stabilising testsuite jira) > >> >> Cheers, >> -- >> Galder Zamarreño >> Sr. Software Engineer >> Infinispan, JBoss Cache >> >> >> ___ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?
Going back to your original question Galder, the exception is most likely thrown because of this sequence of events: 0. Given a cluster {A, B}, a key k and a node C joining. 1. Put acquires the "transaction lock" on node A (blocking rehashing) 2. Put acquires lock for key k on node A 3. Rehashing starts on node B, blocking transactions 4. Put tries to acquire transaction lock on node B Since it's impossible to finish rehashing while the put operation keeps the transaction lock on node A, the best option was to kill the put operation by throwing a RehashInProgressException. I was thinking in the context of transactions when I wrote this code (see https://github.com/danberindei/infinispan/commit/6ed94d3b2e184d4a48d4e781db8d404baf5915a3, this scenario became just a footnote to the generic case with multiple caches), but the scenario also occurs without transactions. Actually I renamed it "state transfer lock" and I moved it to a separate interceptor in my ISPN-1194 branch. Maybe the locking changes in 5.1 will eliminate this scenario, but otherwise we could improve the user experience by retrying the command after the rehashing finishes. Dan On Tue, Sep 13, 2011 at 8:11 PM, Galder Zamarreño wrote: > > On Sep 13, 2011, at 6:38 PM, Mircea Markus wrote: > >> >> On 13 Sep 2011, at 17:22, Galder Zamarreño wrote: >> >>> Hi, >>> >>> I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why >>> "org.infinispan.distribution.RehashInProgressException: Timed out waiting >>> for the transaction lock" is thrown? >>> >>> This is thrown DistTxInterceptor which is added by the >>> InterceptorChainFactory: >>> >>> if (configuration.getCacheMode().isDistributed()) >>> >>> interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class)); >>> else >>> >>> interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class)); >>> >>> However, this is a Hot Rod test and the cache is not configured with a >>> transaction manager, so is DistTxInterceptor really needed? >>> >>> In fact, why a TxInterceptor if no transaction manager is configured? (this >>> kinda goes back to Mircea's email earlier today about a transactions >>> enabled/disabled flag) >> All valid concerns, IMO. >> 5.1 will clarify this by not adding Tx interceptors for non tx caches. > > Is this captured in some other JIRA? I'd like to reference the test and the > JIRA from ISPN-1123 (stabilising testsuite jira) > >> >>> >>> Cheers, >>> -- >>> Galder Zamarreño >>> Sr. Software Engineer >>> Infinispan, JBoss Cache >>> >>> >>> ___ >>> infinispan-dev mailing list >>> infinispan-dev@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> ___ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?
On Sep 14, 2011, at 11:40 AM, Dan Berindei wrote: > Going back to your original question Galder, the exception is most > likely thrown because of this sequence of events: > > 0. Given a cluster {A, B}, a key k and a node C joining. > 1. Put acquires the "transaction lock" on node A (blocking rehashing) > 2. Put acquires lock for key k on node A > 3. Rehashing starts on node B, blocking transactions > 4. Put tries to acquire transaction lock on node B > > Since it's impossible to finish rehashing while the put operation > keeps the transaction lock on node A, the best option was to kill the > put operation by throwing a RehashInProgressException. > > I was thinking in the context of transactions when I wrote this code > (see > https://github.com/danberindei/infinispan/commit/6ed94d3b2e184d4a48d4e781db8d404baf5915a3, > this scenario became just a footnote to the generic case with multiple > caches), but the scenario also occurs without transactions. Actually I > renamed it "state transfer lock" and I moved it to a separate > interceptor in my ISPN-1194 branch. Right, but it shouldn't happen when transactions are off, right? > > Maybe the locking changes in 5.1 will eliminate this scenario, but > otherwise we could improve the user experience by retrying the command > after the rehashing finishes. I'd prefer that, otherwise users are likely to code similar logic which is not nice. > > Dan > > > On Tue, Sep 13, 2011 at 8:11 PM, Galder Zamarreño wrote: >> >> On Sep 13, 2011, at 6:38 PM, Mircea Markus wrote: >> >>> >>> On 13 Sep 2011, at 17:22, Galder Zamarreño wrote: >>> Hi, I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why "org.infinispan.distribution.RehashInProgressException: Timed out waiting for the transaction lock" is thrown? This is thrown DistTxInterceptor which is added by the InterceptorChainFactory: if (configuration.getCacheMode().isDistributed()) interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class)); else interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class)); However, this is a Hot Rod test and the cache is not configured with a transaction manager, so is DistTxInterceptor really needed? In fact, why a TxInterceptor if no transaction manager is configured? (this kinda goes back to Mircea's email earlier today about a transactions enabled/disabled flag) >>> All valid concerns, IMO. >>> 5.1 will clarify this by not adding Tx interceptors for non tx caches. >> >> Is this captured in some other JIRA? I'd like to reference the test and the >> JIRA from ISPN-1123 (stabilising testsuite jira) >> >>> Cheers, -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> ___ >>> infinispan-dev mailing list >>> infinispan-dev@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Galder Zamarreño >> Sr. Software Engineer >> Infinispan, JBoss Cache >> >> >> ___ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?
On Thu, Sep 15, 2011 at 12:25 PM, Galder Zamarreño wrote: > > On Sep 14, 2011, at 11:40 AM, Dan Berindei wrote: > >> Going back to your original question Galder, the exception is most >> likely thrown because of this sequence of events: >> >> 0. Given a cluster {A, B}, a key k and a node C joining. >> 1. Put acquires the "transaction lock" on node A (blocking rehashing) >> 2. Put acquires lock for key k on node A >> 3. Rehashing starts on node B, blocking transactions >> 4. Put tries to acquire transaction lock on node B >> >> Since it's impossible to finish rehashing while the put operation >> keeps the transaction lock on node A, the best option was to kill the >> put operation by throwing a RehashInProgressException. >> >> I was thinking in the context of transactions when I wrote this code >> (see >> https://github.com/danberindei/infinispan/commit/6ed94d3b2e184d4a48d4e781db8d404baf5915a3, >> this scenario became just a footnote to the generic case with multiple >> caches), but the scenario also occurs without transactions. Actually I >> renamed it "state transfer lock" and I moved it to a separate >> interceptor in my ISPN-1194 branch. > > Right, but it shouldn't happen when transactions are off, right? > It will still happen with transactions off, because state transfer will acquire the state transfer lock in exclusive mode and WriteCommands will acquire the state transfer lock in shared mode instead of PrepareCommands with transactions on. If we don't acquire the state transfer lock than we run the risk of pushing stale data to other nodes, are you saying that without transactions enabled this risk is acceptable? >> >> Maybe the locking changes in 5.1 will eliminate this scenario, but >> otherwise we could improve the user experience by retrying the command >> after the rehashing finishes. > > I'd prefer that, otherwise users are likely to code similar logic which is > not nice. > After looking at the scenario again it became clear that step 2. is not necessary at all and the locking enhancements will not change anything. We could change DistributionInterceptor to not hold the state transfer lock during remote calls, which will allow rehashing to proceed on the origin. But then the ownership of the keys might change during the call, so we'll need a retry phase anyway. Dan ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?
On Sep 15, 2011, at 1:31 PM, Dan Berindei wrote: > On Thu, Sep 15, 2011 at 12:25 PM, Galder Zamarreño wrote: >> >> On Sep 14, 2011, at 11:40 AM, Dan Berindei wrote: >> >>> Going back to your original question Galder, the exception is most >>> likely thrown because of this sequence of events: >>> >>> 0. Given a cluster {A, B}, a key k and a node C joining. >>> 1. Put acquires the "transaction lock" on node A (blocking rehashing) >>> 2. Put acquires lock for key k on node A >>> 3. Rehashing starts on node B, blocking transactions >>> 4. Put tries to acquire transaction lock on node B >>> >>> Since it's impossible to finish rehashing while the put operation >>> keeps the transaction lock on node A, the best option was to kill the >>> put operation by throwing a RehashInProgressException. >>> >>> I was thinking in the context of transactions when I wrote this code >>> (see >>> https://github.com/danberindei/infinispan/commit/6ed94d3b2e184d4a48d4e781db8d404baf5915a3, >>> this scenario became just a footnote to the generic case with multiple >>> caches), but the scenario also occurs without transactions. Actually I >>> renamed it "state transfer lock" and I moved it to a separate >>> interceptor in my ISPN-1194 branch. >> >> Right, but it shouldn't happen when transactions are off, right? >> > > It will still happen with transactions off, because state transfer > will acquire the state transfer lock in exclusive mode and > WriteCommands will acquire the state transfer lock in shared mode > instead of PrepareCommands with transactions on. > > If we don't acquire the state transfer lock than we run the risk of > pushing stale data to other nodes, are you saying that without > transactions enabled this risk is acceptable? Hmmm, at first glance no, it does not look acceptable. Refactoring though, as you did, is in order though to move it somewhere else, cos clearly it's logic that's required by both transactional and non-transactional caches. > >>> >>> Maybe the locking changes in 5.1 will eliminate this scenario, but >>> otherwise we could improve the user experience by retrying the command >>> after the rehashing finishes. >> >> I'd prefer that, otherwise users are likely to code similar logic which is >> not nice. >> > > After looking at the scenario again it became clear that step 2. is > not necessary at all and the locking enhancements will not change > anything. > > We could change DistributionInterceptor to not hold the state transfer > lock during remote calls, which will allow rehashing to proceed on the > origin. But then the ownership of the keys might change during the > call, so we'll need a retry phase anyway. > > Dan > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev