On Jul 30, 2014, at 3:27 PM, Jay Pipes <jaypi...@gmail.com> wrote: > It's not about distributed locking. It's about allowing multiple threads to > make some sort of progress in the face of a contentious piece of code.
Sure, multiple threads, multiple process, multiple remote/distributed/cooperating process (all pretty much the same imho, at least at a high-level); all trying to get there work done (like good little worker bees). > Obstruction and lock-free algorithms are preferred, IMO, over lock-based > solutions that sit there and block while something else is doing something. Understandable, preferring the ones that can move forward under contention is great; just experience and knowledge and people that have implemented them know that lock-free solutions are usually multiple of times harder to get right and reason about than just sitting there and blocking. Both can work, I'm just more of a fan of making it work right (keyword `right`) first (using whichever solution is fine with the neutron folks), then coming back and making it work better as needed. > > And yeah, I'm using the term lock-free here, when in fact, there is the > possibility of a lock being held for a very short amount of time in the > low-level storage engine code... > > That said, I completely agree with you on using existing technologies and not > reinventing wheels where appropriate. If I needed a distributed lock, I > wouldn't reinvent one in Python ;) +2 > > Best > -jay I'll go back to being quiet now, :-P -Josh > > On 07/30/2014 03:16 PM, Joshua Harlow wrote: >> I'll just start by saying I'm not the expert in what should-be the >> solution for neutron here (this is their developers ultimate >> decision) but I just wanted to add my thoughts.... >> >> Jays solution looks/sounds like a spin lock with a test-and-set[1] >> (imho still a lock, no matter the makeup u put on it), >> >> Seems similar to: https://review.openstack.org/#/c/97059/ in concept >> that I also saw recently. >> >> I though start to feel we should be figuring out how to use a proven >> correct locking mechanism (kazoo -> zookeeper, tooz -> memcache, >> redis or zookeeper...) and avoid the premature optimization that we >> seem to be falling into when creating our own types of spin locks, >> optimistic locks and so on... I'd much rather prefer correctness that >> *might* be a little slower than a solution that is hard to debug, >> hard to reason about and requires retry magic numbers/hacks (for >> example that prior keystone review has a magic 10 iteration limit, >> after all who really knows what that magic number should be...), >> especially in cases where it is really necessary (I can't qualify to >> say whether this neutron situation is appropriate for this). >> >> Maybe this is the appropriate time to focus on correct (maybe slower, >> maybe requires zookeeper or redis...) solutions instead of reinvent >> another solution that we will regret in the future. I'd rather not >> put my operators through hell (they will be the ones left at the >> middle of the night trying to figure out why the lock didn't lock) >> when I can avoid it... >> >> Just my 2 cents, >> >> [1] http://en.wikipedia.org/wiki/Test-and-set >> >> -Josh >> >> On Jul 30, 2014, at 1:53 PM, Jay Pipes <jaypi...@gmail.com> wrote: >> >>> On 07/30/2014 12:21 PM, Kevin Benton wrote: >>>> Maybe I misunderstood your approach then. >>>> >>>> I though you were suggesting where a node performs an "UPDATE >>>> record WHERE record = last_state_node_saw" query and then checks >>>> the number of affected rows. That's optimistic locking by every >>>> definition I've heard of it. It matches the following statement >>>> from the wiki article you linked to as well: >>>> >>>> "The latter situation (optimistic locking) is only appropriate >>>> when there is less chance of someone needing to access the record >>>> while it is locked; otherwise it cannot be certain that the >>>> update will succeed because the attempt to update the record will >>>> fail if another user updates the record first." >>>> >>>> Did I misinterpret how your approach works? >>> >>> The record is never "locked" in my approach, why is why I don't >>> like to think of it as optimistic locking. It's more like >>> "optimistic read and update with retry if certain conditions >>> continue to be met..." :) >>> >>> To be very precise, the record is never locked explicitly -- either >>> through the use of SELECT FOR UPDATE or some explicit file or >>> distributed lock. InnoDB won't even hold a lock on anything, as it >>> will simply add a new version to the row using its MGCC (sometimes >>> called MVCC) methods. >>> >>> The technique I am showing in the patch relies on the behaviour of >>> the SQL UPDATE statement with a WHERE clause that contains certain >>> columns and values from the original view of the record. The >>> behaviour of the UPDATE statement will be a NOOP when some other >>> thread has updated the record in between the time that the first >>> thread read the record, and the time the first thread attempted to >>> update the record. The caller of UPDATE can detect this NOOP by >>> checking the number of affected rows, and retry the UPDATE if >>> certain conditions remain kosher... >>> >>> So, there's actually no locks taken in the entire process, which is >>> why I object to the term optimistic locking :) I think where the >>> confusion has been is that the initial SELECT and the following >>> UPDATE statements are *not* done in the context of a single SQL >>> transaction... >>> >>> Best, -jay >>> >>>> On Wed, Jul 30, 2014 at 11:07 AM, Jay Pipes <jaypi...@gmail.com >>>> <mailto:jaypi...@gmail.com>> wrote: >>>> >>>> On 07/30/2014 10:53 AM, Kevin Benton wrote: >>>> >>>> Using the UPDATE WHERE statement you described is referred to as >>>> optimistic locking. [1] >>>> >>>> https://docs.jboss.org/__jbossas/docs/Server___Configuration_Guide/4/html/__The_CMP_Engine-Optimistic___Locking.html >>>> >>>> > <https://docs.jboss.org/jbossas/docs/Server_Configuration_Guide/4/html/The_CMP_Engine-Optimistic_Locking.html> >>>> >>>> >>>> SQL != JBoss. >>>> >>>> It's not optimistic locking in the database world. In the >>>> database world, optimistic locking is an entirely separate >>>> animal: >>>> >>>> http://en.wikipedia.org/wiki/__Lock_(database) >>>> <http://en.wikipedia.org/wiki/Lock_(database)> >>>> >>>> And what I am describing is not optimistic lock concurrency in >>>> databases. >>>> >>>> -jay >>>> >>>> >>>> >>>> _________________________________________________ OpenStack-dev >>>> mailing list OpenStack-dev@lists.openstack.__org >>>> <mailto:OpenStack-dev@lists.openstack.org> >>>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev >>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >>>> >>>> >>>> >>>> >>>> >>>> > -- >>>> Kevin Benton >>>> >>>> >>>> _______________________________________________ OpenStack-dev >>>> mailing list OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>> >>> >>>> > _______________________________________________ >>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> _______________________________________________ OpenStack-dev mailing >> list OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev