Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Joshua Harlow]

On Jul 30, 2014, at 3:27 PM, Jay Pipes  wrote:

> It's not about distributed locking. It's about allowing multiple threads to 
> make some sort of progress in the face of a contentious piece of code. 

Sure, multiple threads, multiple process, multiple 
remote/distributed/cooperating process (all pretty much the same imho, at least 
at a high-level); all trying to get there work done (like good little worker 
bees).

> Obstruction and lock-free algorithms are preferred, IMO, over lock-based 
> solutions that sit there and block while something else is doing something.

Understandable, preferring the ones that can move forward under contention is 
great; just experience and knowledge and people that have implemented them know 
that lock-free solutions are usually multiple of times harder to get right and 
reason about than just sitting there and blocking. Both can work, I'm just more 
of a fan of making it work right (keyword `right`) first (using whichever 
solution is fine with the neutron folks), then coming back and making it work 
better as needed.

> 
> And yeah, I'm using the term lock-free here, when in fact, there is the 
> possibility of a lock being held for a very short amount of time in the 
> low-level storage engine code...
> 
> That said, I completely agree with you on using existing technologies and not 
> reinventing wheels where appropriate. If I needed a distributed lock, I 
> wouldn't reinvent one in Python ;)

+2

> 
> Best
> -jay

I'll go back to being quiet now,

:-P

-Josh

> 
> On 07/30/2014 03:16 PM, Joshua Harlow wrote:
>> I'll just start by saying I'm not the expert in what should-be the
>> solution for neutron here (this is their developers ultimate
>> decision) but I just wanted to add my thoughts
>> 
>> Jays solution looks/sounds like a spin lock with a test-and-set[1]
>> (imho still a lock, no matter the makeup u put on it),
>> 
>> Seems similar to: https://review.openstack.org/#/c/97059/ in concept
>> that I also saw recently.
>> 
>> I though start to feel we should be figuring out how to use a proven
>> correct locking mechanism (kazoo -> zookeeper, tooz -> memcache,
>> redis or zookeeper...) and avoid the premature optimization that we
>> seem to be falling into when creating our own types of spin locks,
>> optimistic locks and so on... I'd much rather prefer correctness that
>> *might* be a little slower than a solution that is hard to debug,
>> hard to reason about and requires retry magic numbers/hacks (for
>> example that prior keystone review has a magic 10 iteration limit,
>> after all who really knows what that magic number should be...),
>> especially in cases where it is really necessary (I can't qualify to
>> say whether this neutron situation is appropriate for this).
>> 
>> Maybe this is the appropriate time to focus on correct (maybe slower,
>> maybe requires zookeeper or redis...) solutions instead of reinvent
>> another solution that we will regret in the future. I'd rather not
>> put my operators through hell (they will be the ones left at the
>> middle of the night trying to figure out why the lock didn't lock)
>> when I can avoid it...
>> 
>> Just my 2 cents,
>> 
>> [1] http://en.wikipedia.org/wiki/Test-and-set
>> 
>> -Josh
>> 
>> On Jul 30, 2014, at 1:53 PM, Jay Pipes  wrote:
>> 
>>> On 07/30/2014 12:21 PM, Kevin Benton wrote:
 Maybe I misunderstood your approach then.
 
 I though you were suggesting where a node performs an "UPDATE
 record WHERE record = last_state_node_saw" query and then checks
 the number of affected rows. That's optimistic locking by every
 definition I've heard of it. It matches the following statement
 from the wiki article you linked to as well:
 
 "The latter situation (optimistic locking) is only appropriate
 when there is less chance of someone needing to access the record
 while it is locked; otherwise it cannot be certain that the
 update will succeed because the attempt to update the record will
 fail if another user updates the record first."
 
 Did I misinterpret how your approach works?
>>> 
>>> The record is never "locked" in my approach, why is why I don't
>>> like to think of it as optimistic locking. It's more like
>>> "optimistic read and update with retry if certain conditions
>>> continue to be met..." :)
>>> 
>>> To be very precise, the record is never locked explicitly -- either
>>> through the use of SELECT FOR UPDATE or some explicit file or
>>> distributed lock. InnoDB won't even hold a lock on anything, as it
>>> will simply add a new version to the row using its MGCC (sometimes
>>> called MVCC) methods.
>>> 
>>> The technique I am showing in the patch relies on the behaviour of
>>> the SQL UPDATE statement with a WHERE clause that contains certain
>>> columns and values from the original view of the record. The
>>> behaviour of the UPDATE statement will be a NOOP when some other
>>> thread has updated the record in between the time that the fi

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Jay Pipes]

It's not about distributed locking. It's about allowing multiple threads 
to make some sort of progress in the face of a contentious piece of 
code. Obstruction and lock-free algorithms are preferred, IMO, over 
lock-based solutions that sit there and block while something else is 
doing something.


And yeah, I'm using the term lock-free here, when in fact, there is the 
possibility of a lock being held for a very short amount of time in the 
low-level storage engine code...


That said, I completely agree with you on using existing technologies 
and not reinventing wheels where appropriate. If I needed a distributed 
lock, I wouldn't reinvent one in Python ;)


Best
-jay

On 07/30/2014 03:16 PM, Joshua Harlow wrote:

I'll just start by saying I'm not the expert in what should-be the
solution for neutron here (this is their developers ultimate
decision) but I just wanted to add my thoughts

Jays solution looks/sounds like a spin lock with a test-and-set[1]
(imho still a lock, no matter the makeup u put on it),

Seems similar to: https://review.openstack.org/#/c/97059/ in concept
that I also saw recently.

I though start to feel we should be figuring out how to use a proven
correct locking mechanism (kazoo -> zookeeper, tooz -> memcache,
redis or zookeeper...) and avoid the premature optimization that we
seem to be falling into when creating our own types of spin locks,
optimistic locks and so on... I'd much rather prefer correctness that
*might* be a little slower than a solution that is hard to debug,
hard to reason about and requires retry magic numbers/hacks (for
example that prior keystone review has a magic 10 iteration limit,
after all who really knows what that magic number should be...),
especially in cases where it is really necessary (I can't qualify to
say whether this neutron situation is appropriate for this).

Maybe this is the appropriate time to focus on correct (maybe slower,
maybe requires zookeeper or redis...) solutions instead of reinvent
another solution that we will regret in the future. I'd rather not
put my operators through hell (they will be the ones left at the
middle of the night trying to figure out why the lock didn't lock)
when I can avoid it...

Just my 2 cents,

[1] http://en.wikipedia.org/wiki/Test-and-set

-Josh

On Jul 30, 2014, at 1:53 PM, Jay Pipes  wrote:


On 07/30/2014 12:21 PM, Kevin Benton wrote:

Maybe I misunderstood your approach then.

I though you were suggesting where a node performs an "UPDATE
record WHERE record = last_state_node_saw" query and then checks
the number of affected rows. That's optimistic locking by every
definition I've heard of it. It matches the following statement
from the wiki article you linked to as well:

"The latter situation (optimistic locking) is only appropriate
when there is less chance of someone needing to access the record
while it is locked; otherwise it cannot be certain that the
update will succeed because the attempt to update the record will
fail if another user updates the record first."

Did I misinterpret how your approach works?


The record is never "locked" in my approach, why is why I don't
like to think of it as optimistic locking. It's more like
"optimistic read and update with retry if certain conditions
continue to be met..." :)

To be very precise, the record is never locked explicitly -- either
through the use of SELECT FOR UPDATE or some explicit file or
distributed lock. InnoDB won't even hold a lock on anything, as it
will simply add a new version to the row using its MGCC (sometimes
called MVCC) methods.

The technique I am showing in the patch relies on the behaviour of
the SQL UPDATE statement with a WHERE clause that contains certain
columns and values from the original view of the record. The
behaviour of the UPDATE statement will be a NOOP when some other
thread has updated the record in between the time that the first
thread read the record, and the time the first thread attempted to
update the record. The caller of UPDATE can detect this NOOP by
checking the number of affected rows, and retry the UPDATE if
certain conditions remain kosher...

So, there's actually no locks taken in the entire process, which is
why I object to the term optimistic locking :) I think where the
confusion has been is that the initial SELECT and the following
UPDATE statements are *not* done in the context of a single SQL
transaction...

Best, -jay


On Wed, Jul 30, 2014 at 11:07 AM, Jay Pipes mailto:jaypi...@gmail.com>> wrote:

On 07/30/2014 10:53 AM, Kevin Benton wrote:

Using the UPDATE WHERE statement you described is referred to as
optimistic locking. [1]

https://docs.jboss.org/__jbossas/docs/Server___Configuration_Guide/4/html/__The_CMP_Engine-Optimistic___Locking.html







SQL != JBoss.

It's not optimistic locking in the database world. In the
database world, optimistic locking is an entirely sep

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Joshua Harlow]

I'll just start by saying I'm not the expert in what should-be the solution for 
neutron here (this is their developers ultimate decision) but I just wanted to 
add my thoughts

Jays solution looks/sounds like a spin lock with a test-and-set[1] (imho still 
a lock, no matter the makeup u put on it),

Seems similar to: https://review.openstack.org/#/c/97059/ in concept that I 
also saw recently.

I though start to feel we should be figuring out how to use a proven correct 
locking mechanism (kazoo -> zookeeper, tooz -> memcache, redis or zookeeper...) 
and avoid the premature optimization that we seem to be falling into when 
creating our own types of spin locks, optimistic locks and so on... I'd much 
rather prefer correctness that *might* be a little slower than a solution that 
is hard to debug, hard to reason about and requires retry magic numbers/hacks 
(for example that prior keystone review has a magic 10 iteration limit, after 
all who really knows what that magic number should be...), especially in cases 
where it is really necessary (I can't qualify to say whether this neutron 
situation is appropriate for this).

Maybe this is the appropriate time to focus on correct (maybe slower, maybe 
requires zookeeper or redis...) solutions instead of reinvent another solution 
that we will regret in the future. I'd rather not put my operators through hell 
(they will be the ones left at the middle of the night trying to figure out why 
the lock didn't lock) when I can avoid it...

Just my 2 cents,

[1] http://en.wikipedia.org/wiki/Test-and-set

-Josh

On Jul 30, 2014, at 1:53 PM, Jay Pipes  wrote:

> On 07/30/2014 12:21 PM, Kevin Benton wrote:
>> Maybe I misunderstood your approach then.
>> 
>> I though you were suggesting where a node performs an "UPDATE record
>> WHERE record = last_state_node_saw" query and then checks the number of
>> affected rows. That's optimistic locking by every definition I've heard
>> of it. It matches the following statement from the wiki article you
>> linked to as well:
>> 
>> "The latter situation (optimistic locking) is only appropriate when
>> there is less chance of someone needing to access the record while it is
>> locked; otherwise it cannot be certain that the update will succeed
>> because the attempt to update the record will fail if another user
>> updates the record first."
>> 
>> Did I misinterpret how your approach works?
> 
> The record is never "locked" in my approach, why is why I don't like to think 
> of it as optimistic locking. It's more like "optimistic read and update with 
> retry if certain conditions continue to be met..." :)
> 
> To be very precise, the record is never locked explicitly -- either through 
> the use of SELECT FOR UPDATE or some explicit file or distributed lock. 
> InnoDB won't even hold a lock on anything, as it will simply add a new 
> version to the row using its MGCC (sometimes called MVCC) methods.
> 
> The technique I am showing in the patch relies on the behaviour of the SQL 
> UPDATE statement with a WHERE clause that contains certain columns and values 
> from the original view of the record. The behaviour of the UPDATE statement 
> will be a NOOP when some other thread has updated the record in between the 
> time that the first thread read the record, and the time the first thread 
> attempted to update the record. The caller of UPDATE can detect this NOOP by 
> checking the number of affected rows, and retry the UPDATE if certain 
> conditions remain kosher...
> 
> So, there's actually no locks taken in the entire process, which is why I 
> object to the term optimistic locking :) I think where the confusion has been 
> is that the initial SELECT and the following UPDATE statements are *not* done 
> in the context of a single SQL transaction...
> 
> Best,
> -jay
> 
>> On Wed, Jul 30, 2014 at 11:07 AM, Jay Pipes > > wrote:
>> 
>>On 07/30/2014 10:53 AM, Kevin Benton wrote:
>> 
>>Using the UPDATE WHERE statement you described is referred to as
>>optimistic locking. [1]
>> 
>>
>> https://docs.jboss.org/__jbossas/docs/Server___Configuration_Guide/4/html/__The_CMP_Engine-Optimistic___Locking.html
>>
>> 
>> 
>> 
>>SQL != JBoss.
>> 
>>It's not optimistic locking in the database world. In the database
>>world, optimistic locking is an entirely separate animal:
>> 
>>http://en.wikipedia.org/wiki/__Lock_(database)
>>
>> 
>>And what I am describing is not optimistic lock concurrency in
>>databases.
>> 
>>-jay
>> 
>> 
>> 
>>_
>>OpenStack-dev mailing list
>>OpenStack-dev@lists.openstack.__org
>>
>>http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev 
>> 

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Jay Pipes]

On 07/30/2014 02:29 PM, Kevin Benton wrote:

Yes, we are talking about the same thing. I think the term 'optimistic
locking' comes from what happens during the sql transaction. The sql
engine converts a read (the WHERE clause) and update (the UPDATE clause)
operations into an atomic operation.
The atomic guarantee requires an internal lock in the database on the
record after it is found by the WHERE clause but before the UPDATE is
completed to prevent a simultaneous query with the same WHERE clause
from updating the record at the same time.
So the lock (or some serialization strategy) is still there, but it's
just buried deep in the SQL engine internals where they belong. :-)


Gah, let us resolve to agree then :) Yes, a mutex is held on a 
section/page of the write-ahead transaction log for a moment in order to 
guarantee durability, so sure, yes, there is a lock held.


Just not on the record itself :P

-jay


On Jul 30, 2014 2:00 PM, "Jay Pipes" mailto:jaypi...@gmail.com>> wrote:

On 07/30/2014 12:21 PM, Kevin Benton wrote:

Maybe I misunderstood your approach then.

I though you were suggesting where a node performs an "UPDATE record
WHERE record = last_state_node_saw" query and then checks the
number of
affected rows. That's optimistic locking by every definition
I've heard
of it. It matches the following statement from the wiki article you
linked to as well:

"The latter situation (optimistic locking) is only appropriate when
there is less chance of someone needing to access the record
while it is
locked; otherwise it cannot be certain that the update will succeed
because the attempt to update the record will fail if another user
updates the record first."

Did I misinterpret how your approach works?


The record is never "locked" in my approach, why is why I don't like
to think of it as optimistic locking. It's more like "optimistic
read and update with retry if certain conditions continue to be
met..." :)

To be very precise, the record is never locked explicitly -- either
through the use of SELECT FOR UPDATE or some explicit file or
distributed lock. InnoDB won't even hold a lock on anything, as it
will simply add a new version to the row using its MGCC (sometimes
called MVCC) methods.

The technique I am showing in the patch relies on the behaviour of
the SQL UPDATE statement with a WHERE clause that contains certain
columns and values from the original view of the record. The
behaviour of the UPDATE statement will be a NOOP when some other
thread has updated the record in between the time that the first
thread read the record, and the time the first thread attempted to
update the record. The caller of UPDATE can detect this NOOP by
checking the number of affected rows, and retry the UPDATE if
certain conditions remain kosher...

So, there's actually no locks taken in the entire process, which is
why I object to the term optimistic locking :) I think where the
confusion has been is that the initial SELECT and the following
UPDATE statements are *not* done in the context of a single SQL
transaction...

Best,
-jay

On Wed, Jul 30, 2014 at 11:07 AM, Jay Pipes mailto:jaypi...@gmail.com>
>> wrote:

 On 07/30/2014 10:53 AM, Kevin Benton wrote:

 Using the UPDATE WHERE statement you described is
referred to as
 optimistic locking. [1]


https://docs.jboss.org/jbossas/docs/Server_Configuration_Guide/4/html/The_CMP_Engine-Optimistic_Locking.html




>


 SQL != JBoss.

 It's not optimistic locking in the database world. In the
database
 world, optimistic locking is an entirely separate animal:

http://en.wikipedia.org/wiki/Lock_(database)

 >

 And what I am describing is not optimistic lock concurrency in
 databases.

 -jay



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 >

http://lists.openstack.o

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Kevin Benton]

Yes, we are talking about the same thing. I think the term 'optimistic
locking' comes from what happens during the sql transaction. The sql engine
converts a read (the WHERE clause) and update (the UPDATE clause)
operations into an atomic operation.
The atomic guarantee requires an internal lock in the database on the
record after it is found by the WHERE clause but before the UPDATE is
completed to prevent a simultaneous query with the same WHERE clause from
updating the record at the same time.
So the lock (or some serialization strategy) is still there, but it's just
buried deep in the SQL engine internals where they belong. :-)
On Jul 30, 2014 2:00 PM, "Jay Pipes"  wrote:

> On 07/30/2014 12:21 PM, Kevin Benton wrote:
>
>> Maybe I misunderstood your approach then.
>>
>> I though you were suggesting where a node performs an "UPDATE record
>> WHERE record = last_state_node_saw" query and then checks the number of
>> affected rows. That's optimistic locking by every definition I've heard
>> of it. It matches the following statement from the wiki article you
>> linked to as well:
>>
>> "The latter situation (optimistic locking) is only appropriate when
>> there is less chance of someone needing to access the record while it is
>> locked; otherwise it cannot be certain that the update will succeed
>> because the attempt to update the record will fail if another user
>> updates the record first."
>>
>> Did I misinterpret how your approach works?
>>
>
> The record is never "locked" in my approach, why is why I don't like to
> think of it as optimistic locking. It's more like "optimistic read and
> update with retry if certain conditions continue to be met..." :)
>
> To be very precise, the record is never locked explicitly -- either
> through the use of SELECT FOR UPDATE or some explicit file or distributed
> lock. InnoDB won't even hold a lock on anything, as it will simply add a
> new version to the row using its MGCC (sometimes called MVCC) methods.
>
> The technique I am showing in the patch relies on the behaviour of the SQL
> UPDATE statement with a WHERE clause that contains certain columns and
> values from the original view of the record. The behaviour of the UPDATE
> statement will be a NOOP when some other thread has updated the record in
> between the time that the first thread read the record, and the time the
> first thread attempted to update the record. The caller of UPDATE can
> detect this NOOP by checking the number of affected rows, and retry the
> UPDATE if certain conditions remain kosher...
>
> So, there's actually no locks taken in the entire process, which is why I
> object to the term optimistic locking :) I think where the confusion has
> been is that the initial SELECT and the following UPDATE statements are
> *not* done in the context of a single SQL transaction...
>
> Best,
> -jay
>
>  On Wed, Jul 30, 2014 at 11:07 AM, Jay Pipes > > wrote:
>>
>> On 07/30/2014 10:53 AM, Kevin Benton wrote:
>>
>> Using the UPDATE WHERE statement you described is referred to as
>> optimistic locking. [1]
>>
>> https://docs.jboss.org/__jbossas/docs/Server___
>> Configuration_Guide/4/html/__The_CMP_Engine-Optimistic___Locking.html
>> > Configuration_Guide/4/html/The_CMP_Engine-Optimistic_Locking.html>
>>
>>
>> SQL != JBoss.
>>
>> It's not optimistic locking in the database world. In the database
>> world, optimistic locking is an entirely separate animal:
>>
>> http://en.wikipedia.org/wiki/__Lock_(database)
>> 
>>
>> And what I am describing is not optimistic lock concurrency in
>> databases.
>>
>> -jay
>>
>>
>>
>> _
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.__org
>> 
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> 
>>
>>
>>
>>
>> --
>> Kevin Benton
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Eugene Nikanorov]

In fact there are more applications for distributed locking than just
accessing data in database.
One of such use cases is serializing access to devices.
This is what is not yet hardly needed, but will be as we get more service
drivers working with appliances.

It would be great if some existing library could be adopted for it.

Thanks,
Eugene.


On Thu, Jul 31, 2014 at 12:53 AM, Jay Pipes  wrote:

> On 07/30/2014 12:21 PM, Kevin Benton wrote:
>
>> Maybe I misunderstood your approach then.
>>
>> I though you were suggesting where a node performs an "UPDATE record
>> WHERE record = last_state_node_saw" query and then checks the number of
>> affected rows. That's optimistic locking by every definition I've heard
>> of it. It matches the following statement from the wiki article you
>> linked to as well:
>>
>> "The latter situation (optimistic locking) is only appropriate when
>> there is less chance of someone needing to access the record while it is
>> locked; otherwise it cannot be certain that the update will succeed
>> because the attempt to update the record will fail if another user
>> updates the record first."
>>
>> Did I misinterpret how your approach works?
>>
>
> The record is never "locked" in my approach, why is why I don't like to
> think of it as optimistic locking. It's more like "optimistic read and
> update with retry if certain conditions continue to be met..." :)
>
> To be very precise, the record is never locked explicitly -- either
> through the use of SELECT FOR UPDATE or some explicit file or distributed
> lock. InnoDB won't even hold a lock on anything, as it will simply add a
> new version to the row using its MGCC (sometimes called MVCC) methods.
>
> The technique I am showing in the patch relies on the behaviour of the SQL
> UPDATE statement with a WHERE clause that contains certain columns and
> values from the original view of the record. The behaviour of the UPDATE
> statement will be a NOOP when some other thread has updated the record in
> between the time that the first thread read the record, and the time the
> first thread attempted to update the record. The caller of UPDATE can
> detect this NOOP by checking the number of affected rows, and retry the
> UPDATE if certain conditions remain kosher...
>
> So, there's actually no locks taken in the entire process, which is why I
> object to the term optimistic locking :) I think where the confusion has
> been is that the initial SELECT and the following UPDATE statements are
> *not* done in the context of a single SQL transaction...
>
> Best,
> -jay
>
>  On Wed, Jul 30, 2014 at 11:07 AM, Jay Pipes > > wrote:
>>
>> On 07/30/2014 10:53 AM, Kevin Benton wrote:
>>
>> Using the UPDATE WHERE statement you described is referred to as
>> optimistic locking. [1]
>>
>> https://docs.jboss.org/__jbossas/docs/Server___
>> Configuration_Guide/4/html/__The_CMP_Engine-Optimistic___Locking.html
>>
>> > Configuration_Guide/4/html/The_CMP_Engine-Optimistic_Locking.html>
>>
>>
>> SQL != JBoss.
>>
>> It's not optimistic locking in the database world. In the database
>> world, optimistic locking is an entirely separate animal:
>>
>> http://en.wikipedia.org/wiki/__Lock_(database)
>>
>> 
>>
>> And what I am describing is not optimistic lock concurrency in
>> databases.
>>
>> -jay
>>
>>
>>
>> _
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.__org
>> 
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> 
>>
>>
>>
>>
>>
>> --
>> Kevin Benton
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Clint Byrum]

Excerpts from Jay Pipes's message of 2014-07-30 13:53:38 -0700:
> On 07/30/2014 12:21 PM, Kevin Benton wrote:
> > Maybe I misunderstood your approach then.
> >
> > I though you were suggesting where a node performs an "UPDATE record
> > WHERE record = last_state_node_saw" query and then checks the number of
> > affected rows. That's optimistic locking by every definition I've heard
> > of it. It matches the following statement from the wiki article you
> > linked to as well:
> >
> > "The latter situation (optimistic locking) is only appropriate when
> > there is less chance of someone needing to access the record while it is
> > locked; otherwise it cannot be certain that the update will succeed
> > because the attempt to update the record will fail if another user
> > updates the record first."
> >
> > Did I misinterpret how your approach works?
> 
> The record is never "locked" in my approach, why is why I don't like to 
> think of it as optimistic locking. It's more like "optimistic read and 
> update with retry if certain conditions continue to be met..." :)
> 
> To be very precise, the record is never locked explicitly -- either 
> through the use of SELECT FOR UPDATE or some explicit file or 
> distributed lock. InnoDB won't even hold a lock on anything, as it will 
> simply add a new version to the row using its MGCC (sometimes called 
> MVCC) methods.
> 
> The technique I am showing in the patch relies on the behaviour of the 
> SQL UPDATE statement with a WHERE clause that contains certain columns 
> and values from the original view of the record. The behaviour of the 
> UPDATE statement will be a NOOP when some other thread has updated the 
> record in between the time that the first thread read the record, and 
> the time the first thread attempted to update the record. The caller of 
> UPDATE can detect this NOOP by checking the number of affected rows, and 
> retry the UPDATE if certain conditions remain kosher...
> 
> So, there's actually no locks taken in the entire process, which is why 
> I object to the term optimistic locking :) I think where the confusion 
> has been is that the initial SELECT and the following UPDATE statements 
> are *not* done in the context of a single SQL transaction...
> 

This is all true at a low level Jay. But if you're serializing something
outside the DB by using the "doing it" versus "done it" state, it still
acts like a lock.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Jay Pipes]

On 07/30/2014 12:21 PM, Kevin Benton wrote:

Maybe I misunderstood your approach then.

I though you were suggesting where a node performs an "UPDATE record
WHERE record = last_state_node_saw" query and then checks the number of
affected rows. That's optimistic locking by every definition I've heard
of it. It matches the following statement from the wiki article you
linked to as well:

"The latter situation (optimistic locking) is only appropriate when
there is less chance of someone needing to access the record while it is
locked; otherwise it cannot be certain that the update will succeed
because the attempt to update the record will fail if another user
updates the record first."

Did I misinterpret how your approach works?


The record is never "locked" in my approach, why is why I don't like to 
think of it as optimistic locking. It's more like "optimistic read and 
update with retry if certain conditions continue to be met..." :)


To be very precise, the record is never locked explicitly -- either 
through the use of SELECT FOR UPDATE or some explicit file or 
distributed lock. InnoDB won't even hold a lock on anything, as it will 
simply add a new version to the row using its MGCC (sometimes called 
MVCC) methods.


The technique I am showing in the patch relies on the behaviour of the 
SQL UPDATE statement with a WHERE clause that contains certain columns 
and values from the original view of the record. The behaviour of the 
UPDATE statement will be a NOOP when some other thread has updated the 
record in between the time that the first thread read the record, and 
the time the first thread attempted to update the record. The caller of 
UPDATE can detect this NOOP by checking the number of affected rows, and 
retry the UPDATE if certain conditions remain kosher...


So, there's actually no locks taken in the entire process, which is why 
I object to the term optimistic locking :) I think where the confusion 
has been is that the initial SELECT and the following UPDATE statements 
are *not* done in the context of a single SQL transaction...


Best,
-jay


On Wed, Jul 30, 2014 at 11:07 AM, Jay Pipes mailto:jaypi...@gmail.com>> wrote:

On 07/30/2014 10:53 AM, Kevin Benton wrote:

Using the UPDATE WHERE statement you described is referred to as
optimistic locking. [1]


https://docs.jboss.org/__jbossas/docs/Server___Configuration_Guide/4/html/__The_CMP_Engine-Optimistic___Locking.html




SQL != JBoss.

It's not optimistic locking in the database world. In the database
world, optimistic locking is an entirely separate animal:

http://en.wikipedia.org/wiki/__Lock_(database)


And what I am describing is not optimistic lock concurrency in
databases.

-jay



_
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org

http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev 





--
Kevin Benton


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Kevin Benton]

Maybe I misunderstood your approach then.

I though you were suggesting where a node performs an "UPDATE record WHERE
record = last_state_node_saw" query and then checks the number of affected
rows. That's optimistic locking by every definition I've heard of it. It
matches the following statement from the wiki article you linked to as well:

"The latter situation (optimistic locking) is only appropriate when there
is less chance of someone needing to access the record while it is locked;
otherwise it cannot be certain that the update will succeed because the
attempt to update the record will fail if another user updates the record
first."

Did I misinterpret how your approach works?


On Wed, Jul 30, 2014 at 11:07 AM, Jay Pipes  wrote:

> On 07/30/2014 10:53 AM, Kevin Benton wrote:
>
>> Using the UPDATE WHERE statement you described is referred to as
>> optimistic locking. [1]
>>
>> https://docs.jboss.org/jbossas/docs/Server_Configuration_Guide/4/html/
>> The_CMP_Engine-Optimistic_Locking.html
>>
>
> SQL != JBoss.
>
> It's not optimistic locking in the database world. In the database world,
> optimistic locking is an entirely separate animal:
>
> http://en.wikipedia.org/wiki/Lock_(database)
>
> And what I am describing is not optimistic lock concurrency in databases.
>
> -jay
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Jay Pipes]

On 07/30/2014 10:53 AM, Kevin Benton wrote:

Using the UPDATE WHERE statement you described is referred to as
optimistic locking. [1]

https://docs.jboss.org/jbossas/docs/Server_Configuration_Guide/4/html/The_CMP_Engine-Optimistic_Locking.html


SQL != JBoss.

It's not optimistic locking in the database world. In the database 
world, optimistic locking is an entirely separate animal:


http://en.wikipedia.org/wiki/Lock_(database)

And what I am describing is not optimistic lock concurrency in databases.

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[ZZelle]

Hello,


I stop to improve vxlan population and remove SELECT FOR UPDATE[1] because
i am not sure the current approach is the right approach to handle
vxlan/gre tenant pools:

1- Do we really to populate vxlan/gre tenant pools?
  The neutron-server could also choose randomly an vxlan vni in vni_ranges
and tries to allocate it and retries until allocate success.
  I did not verify but mac address allocation should use the same
principle?
  It is efficient if used_vnis is small enough (<50%) compared to
vni_ranges size.
  I am about to propose an update of neutron.plugins.ml2.drivers.helpers[2]
in this direction.

2- Do we need to populate/update vxlan/gre tenant pools on neutron-server
restart?
  A specific command could populate/update them (neutron-db-manage populate
/ neutron-db-populate)


Any thoughts?

[1] https://review.openstack.org/#/c/101982
[2]
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/helpers.py


Cedric
ZZelle@IRC


On Wed, Jul 30, 2014 at 7:30 PM, Jay Pipes  wrote:

> On 07/30/2014 10:05 AM, Kevin Benton wrote:
>
>> i.e. 'optimistic locking' as opposed to the 'pessimistic locking'
>> referenced in the 3rd link of the email starting the thread.
>>
>
> No, there's no locking.
>
>  On Wed, Jul 30, 2014 at 9:55 AM, Jay Pipes > > wrote:
>>
>> On 07/30/2014 09:48 AM, Doug Wiegley wrote:
>>
>> I'd have to look at the Neutron code, but I suspect that a
>> simple
>> strategy of issuing the UPDATE SQL statement with a WHERE
>> condition that
>>
>>
>> I¹m assuming the locking is for serializing code, whereas for
>> what you
>> describe above, is there some reason we wouldn¹t just use a
>> transaction?
>>
>>
>> Because you can't do a transaction from two different threads...
>>
>> The compare and update strategy is for avoiding the use of SELECT
>> FOR UPDATE.
>>
>> Best,
>> -jay
>>
>>
>>
>> _
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.__org
>> 
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> 
>>
>>
>>
>>
>>
>> --
>> Kevin Benton
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Kevin Benton]

Using the UPDATE WHERE statement you described is referred to as optimistic
locking. [1]

https://docs.jboss.org/jbossas/docs/Server_Configuration_Guide/4/html/The_CMP_Engine-Optimistic_Locking.html


On Wed, Jul 30, 2014 at 10:30 AM, Jay Pipes  wrote:

> On 07/30/2014 10:05 AM, Kevin Benton wrote:
>
>> i.e. 'optimistic locking' as opposed to the 'pessimistic locking'
>> referenced in the 3rd link of the email starting the thread.
>>
>
> No, there's no locking.
>
>  On Wed, Jul 30, 2014 at 9:55 AM, Jay Pipes > > wrote:
>>
>> On 07/30/2014 09:48 AM, Doug Wiegley wrote:
>>
>> I'd have to look at the Neutron code, but I suspect that a
>> simple
>> strategy of issuing the UPDATE SQL statement with a WHERE
>> condition that
>>
>>
>> I¹m assuming the locking is for serializing code, whereas for
>> what you
>> describe above, is there some reason we wouldn¹t just use a
>> transaction?
>>
>>
>> Because you can't do a transaction from two different threads...
>>
>> The compare and update strategy is for avoiding the use of SELECT
>> FOR UPDATE.
>>
>> Best,
>> -jay
>>
>>
>>
>> _
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.__org
>> 
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> 
>>
>>
>>
>>
>>
>> --
>> Kevin Benton
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Jay Pipes]

On 07/30/2014 10:05 AM, Kevin Benton wrote:

i.e. 'optimistic locking' as opposed to the 'pessimistic locking'
referenced in the 3rd link of the email starting the thread.


No, there's no locking.


On Wed, Jul 30, 2014 at 9:55 AM, Jay Pipes mailto:jaypi...@gmail.com>> wrote:

On 07/30/2014 09:48 AM, Doug Wiegley wrote:

I'd have to look at the Neutron code, but I suspect that a
simple
strategy of issuing the UPDATE SQL statement with a WHERE
condition that


I¹m assuming the locking is for serializing code, whereas for
what you
describe above, is there some reason we wouldn¹t just use a
transaction?


Because you can't do a transaction from two different threads...

The compare and update strategy is for avoiding the use of SELECT
FOR UPDATE.

Best,
-jay



_
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org

http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev 





--
Kevin Benton


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Morgan Fainberg]

-Original Message-
From: Jay Pipes 
Reply: OpenStack Development Mailing List (not for usage questions) 
>
Date: July 30, 2014 at 09:59:15
To: openstack-dev@lists.openstack.org >
Subject:  Re: [openstack-dev] [neutron] Cross-server locking for neutron server

> On 07/30/2014 09:48 AM, Doug Wiegley wrote:
> >> I'd have to look at the Neutron code, but I suspect that a simple
> >> strategy of issuing the UPDATE SQL statement with a WHERE condition that
> >
> > I¹m assuming the locking is for serializing code, whereas for what you
> > describe above, is there some reason we wouldn¹t just use a transaction?
>  
> Because you can't do a transaction from two different threads...
>  
> The compare and update strategy is for avoiding the use of SELECT FOR
> UPDATE.
>  
> Best,
> -jay


As a quick example of the optimistic locking you describe (UPDATE with WHERE 
clause) you can take a look at the Keystone “consume trust” logic:

https://review.openstack.org/#/c/97059/14/keystone/trust/backends/sql.py

Line 93 does the initial query, an update is performed then on line 108 and 115 
we do the update and check to see how many rows were affected.

Feel free to hit me up if I can help in any way on this.

Cheers,
Morgan 


—
Morgan Fainberg

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Kevin Benton]

i.e. 'optimistic locking' as opposed to the 'pessimistic locking'
referenced in the 3rd link of the email starting the thread.


On Wed, Jul 30, 2014 at 9:55 AM, Jay Pipes  wrote:

> On 07/30/2014 09:48 AM, Doug Wiegley wrote:
>
>> I'd have to look at the Neutron code, but I suspect that a simple
>>> strategy of issuing the UPDATE SQL statement with a WHERE condition that
>>>
>>
>> I¹m assuming the locking is for serializing code, whereas for what you
>> describe above, is there some reason we wouldn¹t just use a transaction?
>>
>
> Because you can't do a transaction from two different threads...
>
> The compare and update strategy is for avoiding the use of SELECT FOR
> UPDATE.
>
> Best,
> -jay
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Clint Byrum]

Excerpts from Doug Wiegley's message of 2014-07-30 09:48:17 -0700:
> > I'd have to look at the Neutron code, but I suspect that a simple
> > strategy of issuing the UPDATE SQL statement with a WHERE condition that
> 
> I¹m assuming the locking is for serializing code, whereas for what you
> describe above, is there some reason we wouldn¹t just use a transaction?
> 

I believe the code in question is doing something like this:

1) Check DB for initialized "SDN controller driver"
2) Not initialized -> initialize the SDN controller via its API
3) Record in DB that it is initialized

Step (2) above needs serialization, not (3).

Compare and update will end up working like a distributed lock anyway,
because the db model will have to be changed to have an "initializing"
state, and then if initializing fails, you'll have to have a timeout.. and
stealing for stuck processes.

Sometimes a distributed lock is actually a simpler solution.

Tooz will need work, no doubt. Perhaps if we call it 'oslo.locking' it
will make more sense. Anyway, my point stands: trust the experts, avoid
reinventing locking. And if you don't like tooz, extract the locking
code from Heat and turn it into an oslo.locking library or something.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Jay Pipes]

On 07/30/2014 09:48 AM, Doug Wiegley wrote:

I'd have to look at the Neutron code, but I suspect that a simple
strategy of issuing the UPDATE SQL statement with a WHERE condition that


I¹m assuming the locking is for serializing code, whereas for what you
describe above, is there some reason we wouldn¹t just use a transaction?


Because you can't do a transaction from two different threads...

The compare and update strategy is for avoiding the use of SELECT FOR 
UPDATE.


Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Doug Wiegley]

> I'd have to look at the Neutron code, but I suspect that a simple
> strategy of issuing the UPDATE SQL statement with a WHERE condition that

I¹m assuming the locking is for serializing code, whereas for what you
describe above, is there some reason we wouldn¹t just use a transaction?

Thanks,
doug



On 7/30/14, 9:41 AM, "Jay Pipes"  wrote:

>There's also no need to use locks at all for this (distributed or
>otherwise).
>
>You can use a compare and update strategy with an exponential backoff
>similar to the approach taken here:
>
>https://review.openstack.org/#/c/109837/
>
>I'd have to look at the Neutron code, but I suspect that a simple
>strategy of issuing the UPDATE SQL statement with a WHERE condition that
>is constructed to take into account the expected current record state
>would do the trick...
>
>Best,
>-jay
>
>On 07/30/2014 09:33 AM, Clint Byrum wrote:
>> Please do not re-invent locking.. the way we reinvented locking in Heat.
>> ;)
>>
>> There are well known distributed coordination services such as Zookeeper
>> and etcd, and there is an abstraction for them already called tooz:
>>
>> https://git.openstack.org/cgit/stackforge/tooz/
>>
>> Excerpts from Elena Ezhova's message of 2014-07-30 09:09:27 -0700:
>>> Hello everyone!
>>>
>>> Some recent change requests ([1], [2]) show that there is a number of
>>> issues with locking db resources in Neutron.
>>>
>>> One of them is initialization of drivers which can be performed
>>> simultaneously by several neutron servers. In this case locking is
>>> essential for avoiding conflicts which is now mostly done via using
>>> SQLAlchemy's
>>> with_lockmode() method, which emits SELECT..FOR UPDATE resulting in
>>>rows
>>> being locked within a transaction. As it has been already stated by
>>>Mike
>>> Bayer [3], this statement is not supported by Galera and, what¹s more,
>>>by
>>> Postgresql for which a lock doesn¹t work in case when a table is empty.
>>>
>>> That is why there is a need for an easy solution that would allow
>>> cross-server locking and would work for every backend. First thing that
>>> comes into mind is to create a table which would contain all locks
>>>acquired
>>> by various pieces of code. Each time a code, that wishes to access a
>>>table
>>> that needs locking, would have to perform the following steps:
>>>
>>> 1. Check whether a lock is already acquired by using SELECT lock_name
>>>FROM
>>> cross_server_locks table.
>>>
>>> 2. If SELECT returned None, acquire a lock by inserting it into the
>>> cross_server_locks table.
>>>
>>> In other case wait and then try again until a timeout is reached.
>>>
>>> 3. After a code has executed it should release the lock by deleting the
>>> corresponding entry from the cross_server_locks table.
>>>
>>> The locking process can be implemented by decorating a function that
>>> performs a transaction by a special function, or as a context manager.
>>>
>>> Thus, I wanted to ask the community whether this approach deserves
>>> consideration and, if yes, it would be necessary to decide on the
>>>format of
>>> an entry in cross_server_locks table: how a lock_name should be formed,
>>> whether to support different locking modes, etc.
>>>
>>>
>>> [1] https://review.openstack.org/#/c/101982/
>>>
>>> [2] https://review.openstack.org/#/c/107350/
>>>
>>> [3]
>>> 
>>>https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#Pessimistic_Loc
>>>king_-_SELECT_FOR_UPDATE
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Jay Pipes]

There's also no need to use locks at all for this (distributed or 
otherwise).


You can use a compare and update strategy with an exponential backoff 
similar to the approach taken here:


https://review.openstack.org/#/c/109837/

I'd have to look at the Neutron code, but I suspect that a simple 
strategy of issuing the UPDATE SQL statement with a WHERE condition that 
is constructed to take into account the expected current record state 
would do the trick...


Best,
-jay

On 07/30/2014 09:33 AM, Clint Byrum wrote:

Please do not re-invent locking.. the way we reinvented locking in Heat.
;)

There are well known distributed coordination services such as Zookeeper
and etcd, and there is an abstraction for them already called tooz:

https://git.openstack.org/cgit/stackforge/tooz/

Excerpts from Elena Ezhova's message of 2014-07-30 09:09:27 -0700:

Hello everyone!

Some recent change requests ([1], [2]) show that there is a number of
issues with locking db resources in Neutron.

One of them is initialization of drivers which can be performed
simultaneously by several neutron servers. In this case locking is
essential for avoiding conflicts which is now mostly done via using
SQLAlchemy's
with_lockmode() method, which emits SELECT..FOR UPDATE resulting in rows
being locked within a transaction. As it has been already stated by Mike
Bayer [3], this statement is not supported by Galera and, what’s more, by
Postgresql for which a lock doesn’t work in case when a table is empty.

That is why there is a need for an easy solution that would allow
cross-server locking and would work for every backend. First thing that
comes into mind is to create a table which would contain all locks acquired
by various pieces of code. Each time a code, that wishes to access a table
that needs locking, would have to perform the following steps:

1. Check whether a lock is already acquired by using SELECT lock_name FROM
cross_server_locks table.

2. If SELECT returned None, acquire a lock by inserting it into the
cross_server_locks table.

In other case wait and then try again until a timeout is reached.

3. After a code has executed it should release the lock by deleting the
corresponding entry from the cross_server_locks table.

The locking process can be implemented by decorating a function that
performs a transaction by a special function, or as a context manager.

Thus, I wanted to ask the community whether this approach deserves
consideration and, if yes, it would be necessary to decide on the format of
an entry in cross_server_locks table: how a lock_name should be formed,
whether to support different locking modes, etc.


[1] https://review.openstack.org/#/c/101982/

[2] https://review.openstack.org/#/c/107350/

[3]
https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#Pessimistic_Locking_-_SELECT_FOR_UPDATE


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Cross-server locking for neutron server

[Clint Byrum]

Please do not re-invent locking.. the way we reinvented locking in Heat.
;)

There are well known distributed coordination services such as Zookeeper
and etcd, and there is an abstraction for them already called tooz:

https://git.openstack.org/cgit/stackforge/tooz/

Excerpts from Elena Ezhova's message of 2014-07-30 09:09:27 -0700:
> Hello everyone!
> 
> Some recent change requests ([1], [2]) show that there is a number of
> issues with locking db resources in Neutron.
> 
> One of them is initialization of drivers which can be performed
> simultaneously by several neutron servers. In this case locking is
> essential for avoiding conflicts which is now mostly done via using
> SQLAlchemy's
> with_lockmode() method, which emits SELECT..FOR UPDATE resulting in rows
> being locked within a transaction. As it has been already stated by Mike
> Bayer [3], this statement is not supported by Galera and, what’s more, by
> Postgresql for which a lock doesn’t work in case when a table is empty.
> 
> That is why there is a need for an easy solution that would allow
> cross-server locking and would work for every backend. First thing that
> comes into mind is to create a table which would contain all locks acquired
> by various pieces of code. Each time a code, that wishes to access a table
> that needs locking, would have to perform the following steps:
> 
> 1. Check whether a lock is already acquired by using SELECT lock_name FROM
> cross_server_locks table.
> 
> 2. If SELECT returned None, acquire a lock by inserting it into the
> cross_server_locks table.
> 
>In other case wait and then try again until a timeout is reached.
> 
> 3. After a code has executed it should release the lock by deleting the
> corresponding entry from the cross_server_locks table.
> 
> The locking process can be implemented by decorating a function that
> performs a transaction by a special function, or as a context manager.
> 
> Thus, I wanted to ask the community whether this approach deserves
> consideration and, if yes, it would be necessary to decide on the format of
> an entry in cross_server_locks table: how a lock_name should be formed,
> whether to support different locking modes, etc.
> 
> 
> [1] https://review.openstack.org/#/c/101982/
> 
> [2] https://review.openstack.org/#/c/107350/
> 
> [3]
> https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#Pessimistic_Locking_-_SELECT_FOR_UPDATE

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Cross-server locking for neutron server

[Elena Ezhova]

Hello everyone!

Some recent change requests ([1], [2]) show that there is a number of
issues with locking db resources in Neutron.

One of them is initialization of drivers which can be performed
simultaneously by several neutron servers. In this case locking is
essential for avoiding conflicts which is now mostly done via using
SQLAlchemy's
with_lockmode() method, which emits SELECT..FOR UPDATE resulting in rows
being locked within a transaction. As it has been already stated by Mike
Bayer [3], this statement is not supported by Galera and, what’s more, by
Postgresql for which a lock doesn’t work in case when a table is empty.

That is why there is a need for an easy solution that would allow
cross-server locking and would work for every backend. First thing that
comes into mind is to create a table which would contain all locks acquired
by various pieces of code. Each time a code, that wishes to access a table
that needs locking, would have to perform the following steps:

1. Check whether a lock is already acquired by using SELECT lock_name FROM
cross_server_locks table.

2. If SELECT returned None, acquire a lock by inserting it into the
cross_server_locks table.

   In other case wait and then try again until a timeout is reached.

3. After a code has executed it should release the lock by deleting the
corresponding entry from the cross_server_locks table.

The locking process can be implemented by decorating a function that
performs a transaction by a special function, or as a context manager.

Thus, I wanted to ask the community whether this approach deserves
consideration and, if yes, it would be necessary to decide on the format of
an entry in cross_server_locks table: how a lock_name should be formed,
whether to support different locking modes, etc.


[1] https://review.openstack.org/#/c/101982/

[2] https://review.openstack.org/#/c/107350/

[3]
https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#Pessimistic_Locking_-_SELECT_FOR_UPDATE
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev