Re: [infinispan-dev] Keeping track of locked nodes
Thinking some more the old-view approach should happen on C and it doesn't, so we're not covered. The soutions I see here is to either replay the prepare on C as part of the state transfer or not to release the ST lock until the transaction (vs the prepare command) is complete. We've committed to implementing NBST for 5.2, so not releasing the ST lock is not an option. Won't NBST solve this through lock migration? As 5.1.1 ships with blocking state transfer and this fix is for 5.1.1 only I think the solution mentioned above is also worth considering. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
On Fri, Feb 10, 2012 at 12:37 PM, Mircea Markus mmar...@redhat.com wrote: Thinking some more the old-view approach should happen on C and it doesn't, so we're not covered. The soutions I see here is to either replay the prepare on C as part of the state transfer or not to release the ST lock until the transaction (vs the prepare command) is complete. We've committed to implementing NBST for 5.2, so not releasing the ST lock is not an option. Won't NBST solve this through lock migration? As 5.1.1 ships with blocking state transfer and this fix is for 5.1.1 only I think the solution mentioned above is also worth considering. I don't think it's an option for 5.1.1 either: with pessimistic locking, the user controls when the locks are acquired, so the chance of a user transaction block state transfer for a long time is much higher than they are now. For 5.2 I'm going to implement NBST with the current locking scheme first and then add lock migration on top as I think adding both at once will complicate things too much. Cheers Dan ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
One potential problem with this design is when we have a transaction prepared on the old primary, and the new primary owner is a cache that just started. The new cache won't have any prepared transactions, so no backup locks to prevent new transactions from acquiring the lock. I'm pretty sure this issue has come up in our discussions before, but I can't remember how we decided to handle it. I don't rememebr disscussing it but good point. The new owner-node being up means that state transfer is finished. State transfe won't start before the locks on the previous-owner are released, so there shouldn't be any locking issue. Unless I'm wrong :) Mircea, perhaps an update of https://community.jboss.org/wiki/SingleNodeLockingModel is in order? ;-) yes, I''ll update the docs. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
On Thu, Feb 9, 2012 at 3:07 PM, Mircea Markus mmar...@redhat.com wrote: One potential problem with this design is when we have a transaction prepared on the old primary, and the new primary owner is a cache that just started. The new cache won't have any prepared transactions, so no backup locks to prevent new transactions from acquiring the lock. I'm pretty sure this issue has come up in our discussions before, but I can't remember how we decided to handle it. I don't rememebr disscussing it but good point. The new owner-node being up means that state transfer is finished. State transfe won't start before the locks on the previous-owner are released, so there shouldn't be any locking issue. Unless I'm wrong :) State transfer won't start before the prepare command finishes, but it can start in the time between the prepare command and the commit command. So it's entirely possible for prepare to run with one CH/primary owner and for commit to run with a different CH/primary owner. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
- Original Message -
From: Dan Berindei dan.berin...@gmail.com
To: infinispan -Dev List infinispan-dev@lists.jboss.org
Sent: Thursday, February 9, 2012 4:31:14 PM
Subject: Re: [infinispan-dev] Keeping track of locked nodes
On Thu, Feb 9, 2012 at 3:07 PM, Mircea Markus mmar...@redhat.com
wrote:
One potential problem with this design is when we have a
transaction
prepared on the old primary, and the new primary owner is a cache
that
just started. The new cache won't have any prepared transactions,
so
no backup locks to prevent new transactions from acquiring the
lock.
I'm pretty sure this issue has come up in our discussions before,
but
I can't remember how we decided to handle it.
I don't rememebr disscussing it but good point.
The new owner-node being up means that state transfer is finished.
State transfe won't start before the locks on the previous-owner
are released, so there shouldn't be any locking issue. Unless I'm
wrong :)
State transfer won't start before the prepare command finishes, but
it
can start in the time between the prepare command and the commit
command.
So it's entirely possible for prepare to run with one CH/primary
owner
and for commit to run with a different CH/primary owner.
Ah I see. I thought that the state transfer is blocked until the
transaction(v.s. command) completes.
if we have numOwners=2, ch(k)={A,B} and C comes up such that ch(k)={C,B} then
the second prepare won't be able to acquire lock on B, as there's already a
transaction that has prepared in an previous view.
There might still be a problem when numOwner=1 though as concurrent write might
be applied on C, then overwritten by the original transaction's commit. That's
if we don't migrate the prepare state over during state transfer, which I don't
think we do?
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
- Original Message -
From: Mircea Markus mmar...@redhat.com
To: infinispan -Dev List infinispan-dev@lists.jboss.org
Sent: Thursday, February 9, 2012 5:45:08 PM
Subject: Re: [infinispan-dev] Keeping track of locked nodes
- Original Message -
From: Dan Berindei dan.berin...@gmail.com
To: infinispan -Dev List infinispan-dev@lists.jboss.org
Sent: Thursday, February 9, 2012 4:31:14 PM
Subject: Re: [infinispan-dev] Keeping track of locked nodes
On Thu, Feb 9, 2012 at 3:07 PM, Mircea Markus mmar...@redhat.com
wrote:
One potential problem with this design is when we have a
transaction
prepared on the old primary, and the new primary owner is a
cache
that
just started. The new cache won't have any prepared
transactions,
so
no backup locks to prevent new transactions from acquiring the
lock.
I'm pretty sure this issue has come up in our discussions
before,
but
I can't remember how we decided to handle it.
I don't rememebr disscussing it but good point.
The new owner-node being up means that state transfer is
finished.
State transfe won't start before the locks on the previous-owner
are released, so there shouldn't be any locking issue. Unless I'm
wrong :)
State transfer won't start before the prepare command finishes, but
it
can start in the time between the prepare command and the commit
command.
So it's entirely possible for prepare to run with one CH/primary
owner
and for commit to run with a different CH/primary owner.
Ah I see. I thought that the state transfer is blocked until the
transaction(v.s. command) completes.
if we have numOwners=2, ch(k)={A,B} and C comes up such that
ch(k)={C,B} then the second prepare won't be able to acquire lock on
B, as there's already a transaction that has prepared in an previous
view.
Thinking some more the old-view approach should happen on C and it doesn't, so
we're not covered.
The soutions I see here is to either replay the prepare on C as part of the
state transfer or not to release the ST lock until the transaction (vs the
prepare command) is complete.
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
https://issues.jboss.org/browse/ISPN-1857
- Original Message -
From: Mircea Markus mmar...@redhat.com
To: infinispan -Dev List infinispan-dev@lists.jboss.org
Sent: Thursday, February 9, 2012 6:04:59 PM
Subject: Re: [infinispan-dev] Keeping track of locked nodes
- Original Message -
From: Mircea Markus mmar...@redhat.com
To: infinispan -Dev List infinispan-dev@lists.jboss.org
Sent: Thursday, February 9, 2012 5:45:08 PM
Subject: Re: [infinispan-dev] Keeping track of locked nodes
- Original Message -
From: Dan Berindei dan.berin...@gmail.com
To: infinispan -Dev List infinispan-dev@lists.jboss.org
Sent: Thursday, February 9, 2012 4:31:14 PM
Subject: Re: [infinispan-dev] Keeping track of locked nodes
On Thu, Feb 9, 2012 at 3:07 PM, Mircea Markus
mmar...@redhat.com
wrote:
One potential problem with this design is when we have a
transaction
prepared on the old primary, and the new primary owner is a
cache
that
just started. The new cache won't have any prepared
transactions,
so
no backup locks to prevent new transactions from acquiring
the
lock.
I'm pretty sure this issue has come up in our discussions
before,
but
I can't remember how we decided to handle it.
I don't rememebr disscussing it but good point.
The new owner-node being up means that state transfer is
finished.
State transfe won't start before the locks on the
previous-owner
are released, so there shouldn't be any locking issue. Unless
I'm
wrong :)
State transfer won't start before the prepare command finishes,
but
it
can start in the time between the prepare command and the commit
command.
So it's entirely possible for prepare to run with one CH/primary
owner
and for commit to run with a different CH/primary owner.
Ah I see. I thought that the state transfer is blocked until the
transaction(v.s. command) completes.
if we have numOwners=2, ch(k)={A,B} and C comes up such that
ch(k)={C,B} then the second prepare won't be able to acquire lock
on
B, as there's already a transaction that has prepared in an
previous
view.
Thinking some more the old-view approach should happen on C and it
doesn't, so we're not covered.
The soutions I see here is to either replay the prepare on C as part
of the state transfer or not to release the ST lock until the
transaction (vs the prepare command) is complete.
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
On Thu, Feb 9, 2012 at 8:10 PM, Mircea Markus mmar...@redhat.com wrote:
https://issues.jboss.org/browse/ISPN-1857
- Original Message -
From: Mircea Markus mmar...@redhat.com
To: infinispan -Dev List infinispan-dev@lists.jboss.org
Sent: Thursday, February 9, 2012 6:04:59 PM
Subject: Re: [infinispan-dev] Keeping track of locked nodes
- Original Message -
From: Mircea Markus mmar...@redhat.com
To: infinispan -Dev List infinispan-dev@lists.jboss.org
Sent: Thursday, February 9, 2012 5:45:08 PM
Subject: Re: [infinispan-dev] Keeping track of locked nodes
- Original Message -
From: Dan Berindei dan.berin...@gmail.com
To: infinispan -Dev List infinispan-dev@lists.jboss.org
Sent: Thursday, February 9, 2012 4:31:14 PM
Subject: Re: [infinispan-dev] Keeping track of locked nodes
On Thu, Feb 9, 2012 at 3:07 PM, Mircea Markus
mmar...@redhat.com
wrote:
One potential problem with this design is when we have a
transaction
prepared on the old primary, and the new primary owner is a
cache
that
just started. The new cache won't have any prepared
transactions,
so
no backup locks to prevent new transactions from acquiring
the
lock.
I'm pretty sure this issue has come up in our discussions
before,
but
I can't remember how we decided to handle it.
I don't rememebr disscussing it but good point.
The new owner-node being up means that state transfer is
finished.
State transfe won't start before the locks on the
previous-owner
are released, so there shouldn't be any locking issue. Unless
I'm
wrong :)
State transfer won't start before the prepare command finishes,
but
it
can start in the time between the prepare command and the commit
command.
So it's entirely possible for prepare to run with one CH/primary
owner
and for commit to run with a different CH/primary owner.
Ah I see. I thought that the state transfer is blocked until the
transaction(v.s. command) completes.
if we have numOwners=2, ch(k)={A,B} and C comes up such that
ch(k)={C,B} then the second prepare won't be able to acquire lock
on
B, as there's already a transaction that has prepared in an
previous
view.
Thinking some more the old-view approach should happen on C and it
doesn't, so we're not covered.
The soutions I see here is to either replay the prepare on C as part
of the state transfer or not to release the ST lock until the
transaction (vs the prepare command) is complete.
We've committed to implementing NBST for 5.2, so not releasing the ST
lock is not an option.
The only choice then is to transfer the locking information along with
the state.
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
In this particular case it just means that the commit is sync even if it's configured to be async. But there are more places where we check the remote lock nodes list, e.g. BaseRpcInterceptor.shouldInvokeRemotely or AbstractEnlistmentAdapter - which, incidentally, could probably settle with a hasRemoteLocks boolean flag instead of the nodes collection. TransactionXaAdapter.forgetSuccessfullyCompletedTransaction does use it properly when recovery is enabled - if we didn't keep the collection around it would have to compute it again from the list of keys. The same with PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare, but there we're on thefailure path already so recomputing the address set wouldn't hurt that much. I may have missed others... Cheers Dan On Fri, Jan 20, 2012 at 8:52 AM, Manik Surtani ma...@jboss.org wrote: Well, a view change may not mean that the nodes prepared on have changed. But still, there would almost certainly be a better way to do this than a collection of addresses. So the goal is to determine whether the set of nodes the prepare has been sent to and the current set of affected nodes at the time of commit has changed? And what are the consequences of getting this (pessimistically) wrong, just that the prepare gets repeated on some nodes? On 20 Jan 2012, at 03:54, Mircea Markus wrote: On 19 Jan 2012, at 19:22, Sanne Grinovero wrote: I just noticed that org.infinispan.transaction.LocalTransaction is keeping track of Addresses on which locks where acquired. That's surprising me .. why should it ever be interested in the specific Address? I'd expect it to be able to figure that out when needed, especially since the Address owning the lock might change over time I don't understand to track for a specific node. This information is required at least here[1], but not sure wether we cannot rely on the viewId which is now associated with every transaction to replace that logic. [1] http://bit.ly/wgMIHH ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
On 20 Jan 2012, at 09:11, Dan Berindei wrote: In this particular case it just means that the commit is sync even if it's configured to be async. But there are more places where we check the remote lock nodes list, e.g. BaseRpcInterceptor.shouldInvokeRemotely or AbstractEnlistmentAdapter - which, incidentally, could probably settle with a hasRemoteLocks boolean flag instead of the nodes collection. TransactionXaAdapter.forgetSuccessfullyCompletedTransaction does use it properly when recovery is enabled - if we didn't keep the collection around it would have to compute it again from the list of keys. The same with PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare, but there we're on thefailure path already so recomputing the address set wouldn't hurt that much. I may have missed others... As mentioned, these can be determined based on the set of locks held by that tramsaction. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
On 20 Jan 2012, at 09:33, Manik Surtani wrote: On 20 Jan 2012, at 14:41, Dan Berindei wrote: In this particular case it just means that the commit is sync even if it's configured to be async. But there are more places where we check the remote lock nodes list, e.g. BaseRpcInterceptor.shouldInvokeRemotely or AbstractEnlistmentAdapter - which, incidentally, could probably settle with a hasRemoteLocks boolean flag instead of the nodes collection. TransactionXaAdapter.forgetSuccessfullyCompletedTransaction does use it properly when recovery is enabled - if we didn't keep the collection around it would have to compute it again from the list of keys. The same with PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare, but there we're on thefailure path already so recomputing the address set wouldn't hurt that much. I may have missed others… Cool. Mircea, reckon we can patch this quickly and with low risk? Or is it high risk at this stage? I don't think it's a good moment for this right now. I'm not even convinced that this is the way go, as it might be cheaper to cache this information than to calculate it when needed. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
On 20 January 2012 11:49, Mircea Markus mircea.mar...@jboss.com wrote: On 20 Jan 2012, at 09:33, Manik Surtani wrote: On 20 Jan 2012, at 14:41, Dan Berindei wrote: In this particular case it just means that the commit is sync even if it's configured to be async. But there are more places where we check the remote lock nodes list, e.g. BaseRpcInterceptor.shouldInvokeRemotely or AbstractEnlistmentAdapter - which, incidentally, could probably settle with a hasRemoteLocks boolean flag instead of the nodes collection. TransactionXaAdapter.forgetSuccessfullyCompletedTransaction does use it properly when recovery is enabled - if we didn't keep the collection around it would have to compute it again from the list of keys. The same with PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare, but there we're on thefailure path already so recomputing the address set wouldn't hurt that much. I may have missed others… Cool. Mircea, reckon we can patch this quickly and with low risk? Or is it high risk at this stage? I don't think it's a good moment for this right now. I'm not even convinced that this is the way go, as it might be cheaper to cache this information than to calculate it when needed. Just to clarify, I wasn't reporting a contention point or a performance issue. I was just puzzled by the design as it's very different than what I was expecting. I think we should move towards a design for which we don't really consider the locks to be positioned on a specific node, they should be free to move around (still deterministically, I mean on rehash). Not asking for urgent changes! Cheers, Sanne ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
Cool. Mircea, reckon we can patch this quickly and with low risk? Or is it high risk at this stage? I don't think it's a good moment for this right now. I'm not even convinced that this is the way go, as it might be cheaper to cache this information than to calculate it when needed. Just to clarify, I wasn't reporting a contention point or a performance issue. I was just puzzled by the design as it's very different than what I was expecting. I think we should move towards a design for which we don't really consider the locks to be positioned on a specific node, they should be free to move around (still deterministically, I mean on rehash). Not asking for urgent changes! +1, you've been quite convincing about this in Lisbon :-) However ATM the lock failover is mainly managed by the transaction originator and is not migrated across during topology changes. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
On 20 January 2012 12:41, Mircea Markus mircea.mar...@jboss.com wrote: Cool. Mircea, reckon we can patch this quickly and with low risk? Or is it high risk at this stage? I don't think it's a good moment for this right now. I'm not even convinced that this is the way go, as it might be cheaper to cache this information than to calculate it when needed. Just to clarify, I wasn't reporting a contention point or a performance issue. I was just puzzled by the design as it's very different than what I was expecting. I think we should move towards a design for which we don't really consider the locks to be positioned on a specific node, they should be free to move around (still deterministically, I mean on rehash). Not asking for urgent changes! +1, you've been quite convincing about this in Lisbon :-) However ATM the lock failover is mainly managed by the transaction originator and is not migrated across during topology changes. So if locks are failed-over, is your transaction originator able to find the new owner to unlock it? That's the core of my question on keeping the Address.. not performance related. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Keeping track of locked nodes
Well, a view change may not mean that the nodes prepared on have changed. But still, there would almost certainly be a better way to do this than a collection of addresses. So the goal is to determine whether the set of nodes the prepare has been sent to and the current set of affected nodes at the time of commit has changed? And what are the consequences of getting this (pessimistically) wrong, just that the prepare gets repeated on some nodes? On 20 Jan 2012, at 03:54, Mircea Markus wrote: On 19 Jan 2012, at 19:22, Sanne Grinovero wrote: I just noticed that org.infinispan.transaction.LocalTransaction is keeping track of Addresses on which locks where acquired. That's surprising me .. why should it ever be interested in the specific Address? I'd expect it to be able to figure that out when needed, especially since the Address owning the lock might change over time I don't understand to track for a specific node. This information is required at least here[1], but not sure wether we cannot rely on the viewId which is now associated with every transaction to replace that logic. [1] http://bit.ly/wgMIHH ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
