[openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-12 Thread Ilya Chukhnakov 
Hi everyone.

I’ve recently found that straightforward use of NeutronDbObject is prone to
concurrency-related problems.

I’ve submitted a patch set [3] with some tests to show that without special
treatment using NeutronDbObject could lead to unexpected results.

Further patch sets will provide acquire_object/acquire_objects contextmanager
methods to the NeutronDbObject class. These methods are to be used in place of
get_object/get_objects whenever the user intends to make changes to the object.
These methods would start an autonested_transaction.

There are (at least) two potential options for the implementation:

1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).

   pros:
 - the object is guaranteed to not be changed while within the context

   cons:
 - prone to deadlocks ([1] and potentially when locking multiple objects)

2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
   counter [2] or add our own. If conflicting changes are detected upon exiting
   the context (i.e. version counter held differs from the one in the DB), will
   raise OSLO RetryRequest exception.

   pros:
 - does not require locking

   cons:
 - require an additional field in the models

While opt.2 only prevents the conflicting changes, but does not guarantee that
the object does not change while within the context, opt.1 may seem
preferential. But even with opt.1 the user should not expect that the changes
made to the object while within the context will get to the database as the
autonested_transaction could fail on flush/commit.

So I’d like to hear others’ opinion on the problem and which of the two
implementation options would be preferred? Or maybe someone has a better idea.

[1] 
https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
[2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html

[3] https://review.openstack.org/#/c/315705/

--
Thanks,
Ilya
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-13 Thread Ilya Chukhnakov 
>In a multi-writer galera cluster both SELECT FOR UPDATE and CAS won't fail 
>until commit time

Good point, I did not consider multi-writer. But anyway, my point was that 
opt.2 is no worse than opt.1.


> On 13 May 2016, at 12:04, Kevin Benton  wrote:
> 
> >While opt.2 only prevents the conflicting changes, but does not guarantee 
> >that the object does not change while within the context,
> 
> I'm not sure what you mean here. In a multi-writer galera cluster both SELECT 
> FOR UPDATE and CAS won't fail until commit time if someone writes to another 
> server.
> So the DB lock only provides the conflicting change guarantee in a 
> single-writer mysql setup.
> 
> On Thu, May 12, 2016 at 11:26 AM, Ilya Chukhnakov  <mailto:ichukhna...@mirantis.com>> wrote:
> Hi everyone.
> 
> I’ve recently found that straightforward use of NeutronDbObject is prone to
> concurrency-related problems.
> 
> I’ve submitted a patch set [3] with some tests to show that without special
> treatment using NeutronDbObject could lead to unexpected results.
> 
> Further patch sets will provide acquire_object/acquire_objects contextmanager
> methods to the NeutronDbObject class. These methods are to be used in place of
> get_object/get_objects whenever the user intends to make changes to the 
> object.
> These methods would start an autonested_transaction.
> 
> There are (at least) two potential options for the implementation:
> 
> 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).
> 
>pros:
>  - the object is guaranteed to not be changed while within the context
> 
>cons:
>  - prone to deadlocks ([1] and potentially when locking multiple objects)
> 
> 2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
>counter [2] or add our own. If conflicting changes are detected upon 
> exiting
>the context (i.e. version counter held differs from the one in the DB), 
> will
>raise OSLO RetryRequest exception.
> 
>pros:
>  - does not require locking
> 
>cons:
>  - require an additional field in the models
> 
> While opt.2 only prevents the conflicting changes, but does not guarantee that
> the object does not change while within the context, opt.1 may seem
> preferential. But even with opt.1 the user should not expect that the changes
> made to the object while within the context will get to the database as the
> autonested_transaction could fail on flush/commit.
> 
> So I’d like to hear others’ opinion on the problem and which of the two
> implementation options would be preferred? Or maybe someone has a better idea.
> 
> [1] 
> https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
>  
> <https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad>
> [2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html 
> <http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html>
> 
> [3] https://review.openstack.org/#/c/315705/ 
> <https://review.openstack.org/#/c/315705/>
> 
> --
> Thanks,
> Ilya
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> <http://openstack-dev-requ...@lists.openstack.org/?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-13 Thread Ilya Chukhnakov 
> there is ongoing work to add the revision number
Thanks for pointing out. Then I’ll stick with opt.2.

> Then, the object should before update() fetch the current state and try to 
> merge the requested changes to the current state of the object
I don’t think merging states automatically could be possible without knowledge 
of business logic (the ab = a + b test case clearly shows that I believe). So I 
was thinking about throwing the Retry exception and let the user take care of 
that (e.g. with_db_retry).


> On 13 May 2016, at 11:22, Korzeniewski, Artur  
> wrote:
> 
> Hi Ilya,
> Thanks for investigating the concurrency issue. It is indeed a problem from 
> multithreaded neutron-server point of view.
> 
> Regarding the opt.2, there is ongoing work to add the revision number[1] to 
> main neutron resources (port, network, subnet...) This is connected to spec 
> [2] and Launchpad bug [3]: push all object changes as AMQP notifications. I 
> think [1] is implementing part of your idea about version number. If version 
> number will be added to standard attributes table, it will automagically 
> appear as a new field in object.
> 
> What is left to be done in opt.2 is to handle situation when object is trying 
> to update the state and the revision is not the latest. Then, the object 
> should before update() fetch the current state and try to merge the requested 
> changes to the current state of the object, and then apply it in the DB. Also 
> it would send the newest object state via AMQP notification, obsoleting the 
> previous configuration.
> 
> We can also check how nova is handling the concurrency issue in their OVO 
> usage model.
> 
> Regards,
> Artur
> IRC: korzen
> 
> [1] https://review.openstack.org/#/c/303966/11
> [2] https://review.openstack.org/#/c/225995
> [3] https://bugs.launchpad.net/neutron/+bug/1516195
> 
> -Original Message-
> From: Ilya Chukhnakov [mailto:ichukhna...@mirantis.com] 
> Sent: Thursday, May 12, 2016 8:26 PM
> To: OpenStack Development Mailing List 
> Subject: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues
> 
> Hi everyone.
> 
> I’ve recently found that straightforward use of NeutronDbObject is prone to 
> concurrency-related problems.
> 
> I’ve submitted a patch set [3] with some tests to show that without special 
> treatment using NeutronDbObject could lead to unexpected results.
> 
> Further patch sets will provide acquire_object/acquire_objects contextmanager 
> methods to the NeutronDbObject class. These methods are to be used in place 
> of get_object/get_objects whenever the user intends to make changes to the 
> object.
> These methods would start an autonested_transaction.
> 
> There are (at least) two potential options for the implementation:
> 
> 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).
> 
>   pros:
> - the object is guaranteed to not be changed while within the context
> 
>   cons:
> - prone to deadlocks ([1] and potentially when locking multiple objects)
> 
> 2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
>   counter [2] or add our own. If conflicting changes are detected upon exiting
>   the context (i.e. version counter held differs from the one in the DB), will
>   raise OSLO RetryRequest exception.
> 
>   pros:
> - does not require locking
> 
>   cons:
> - require an additional field in the models
> 
> While opt.2 only prevents the conflicting changes, but does not guarantee 
> that the object does not change while within the context, opt.1 may seem 
> preferential. But even with opt.1 the user should not expect that the changes 
> made to the object while within the context will get to the database as the 
> autonested_transaction could fail on flush/commit.
> 
> So I’d like to hear others’ opinion on the problem and which of the two 
> implementation options would be preferred? Or maybe someone has a better idea.
> 
> [1] 
> https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
> [2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html
> 
> [3] https://review.openstack.org/#/c/315705/
> 
> --
> Thanks,
> Ilya
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-16 Thread Ilya Chukhnakov 
On 16 May 2016, at 18:18, Michał Dulko  wrote:
> In Cinder we're handling similar problems related to race conditions
> between status check and status change with conditional updates.
> Basically we're doing "UPDATE table SET status='abc' WHERE id=1 AND
> status='status_allowing_transition_to_foo';".
Thanks for the info. I'll certainly look into it. But for now I'm planning to 
reuse
revision numbers from [1].

> In general the problem you're mentioning is related more to the
> concurrent DB updates and o.vo never aimed for magically solving that
> problem. I believe you've had same problem with raw SQLA objects.
For SQLA we at least have an option to use with_for_update (I've found it is 
being
used in some places). But with OVO we do not have that level of control yet.

[1] https://review.openstack.org/#/c/303966__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-16 Thread Ilya Chukhnakov 

> On 16 May 2016, at 20:01, Michał Dulko  wrote:
> 
> It's not directly related, but this reminds me of tests done by geguileo
> [1] some time ago that were comparing different methods of preventing DB
> race conditions in concurrent environment. Maybe you'll also find them
> useful as you'll probably need to do something like conditional update
> to increment a revision number.
> 
> [1] https://github.com/Akrog/test-cinder-atomic-states 
> 

Thanks for the link. The SQLA revisions are similar to the 
'solutions/update_with_where',
but they use the dedicated column for that [2]. And as long as it is properly 
configured,
it happens 'automagically' (SQLA will take care of adding proper 'where' to 
'update').

[2] http://docs.sqlalchemy.org/en/latest/orm/versioning.html__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Newton blueprints call for action

2016-04-13 Thread Ilya Chukhnakov 
Hello everyone!

Count me in for the VLAN aware VMS. I already have a [seemingly working] 
proof-of-concept for the OVS driver case and expect to submit it for the review 
in a few days.


> On 13 Apr 2016, at 12:51, Oleg Bondarev  wrote:
> 
> 
> -- Forwarded message --
> From: Sukhdev Kapur mailto:sukhdevka...@gmail.com>>
> Date: Mon, Apr 11, 2016 at 7:33 AM
> Subject: Re: [openstack-dev] [Neutron] Newton blueprints call for action
> To: "OpenStack Development Mailing List (not for usage questions)" 
> mailto:openstack-dev@lists.openstack.org>>
> Cc: Bence Romsics  >, yan.songm...@zte.com.cn 
> 
> 
> 
> Hi Rossella, 
> 
> Good to hear that you will follow through with this. Ironic is looking for 
> this API as well for bare metal deployments. We would love to work with you 
> to make sure that this API/Implementation works for all servers ( VMs as well 
> BMs)
> 
> Thanks
> -Sukhdev
> 
> 
> On Wed, Apr 6, 2016 at 4:32 AM, Rossella Sblendido  > wrote:
> 
> 
> On 04/05/2016 05:43 AM, Armando M. wrote:
> >
> > With this email I would like to appeal to the people in CC to report
> > back their interest in continuing working on these items in their
> > respective capacities, and/or the wider community, in case new owners
> > need to be identified.
> >
> > I look forward to hearing back, hoping we can find the right resources
> > to resume progress, and bring these important requirements to completion
> > in time for Newton.
> 
> Count me in for the vlan-aware-vms. We have now a solid design, it's
> only a matter of putting it into code. I will help any way I can, I
> really want to see this feature in Newton.
> 
> cheers,
> 
> Rossella
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 
> 
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kuryr] Spec and devref placement

2016-09-13 Thread Ilya Chukhnakov 
+1 for (b) or (d). devrefs should should be where the related code is


> On 12 Sep 2016, at 18:18, Irena Berezovsky  wrote:
> 
> I am fine with option (b) as well.
> We can add option (d):
> Specs in openstack/kuryr-specs but devrefs in each specific project,
> i.e., the one that will end up with the implementation code.
> 
> 
> On Mon, Sep 12, 2016 at 2:38 PM, Antoni Segura Puimedon  > wrote:
> Hi Kuryrs!
> 
> On September 5th's weekly IRC meeting Irena Berezovsky suggested that
> we should take a decision regarding the location of specs and devrefs.
> 
> Currently we default to putting all the specs and devrefs for:
> - Kuryr
> - Kuryr-libnetwork
> - Kuryr-kubernetes
> 
> to openstack/kuryr. Fuxi is still being integrated and keeps its own doc.
> 
> The three proposals that came up where:
> a) All specs and devrefs to openstack/kuryr
> b) Specs in openstack/kuryr but devrefs in each specific project,
> i.e., the one that will end up with the implementation code.
> c) Both specs and devrefs in each separate Kuryr project.
> 
> I would like to advocate for option (b). It makes things easy for when
> specs involve multiple kuryr pieces and, at the same time, it keeps
> development information in the place where you'd expect, close to the
> code.
> 
> Please, weigh on this issue here in the ML or in the weekly IRC
> meeting today. The idea is to reach a decision by next week's weekly
> IRC meeting and then write it in each subproject's "how to contribute"
> 
> See you later in the weekly IRC,
> 
> Toni
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kuryr] Kuryr Kubernetes Demo

2016-12-18 Thread Ilya Chukhnakov 
Hello, fellow kuryrs!

I'd like to share a demo [1] that shows how Kuryr enables OpenStack Neutron 
networking for Kubernetes and allows Pods to access Nova VMs and vice versa.

[1] https://asciinema.org/a/51qn06lgz78gcbzascpl2du4w 


---
Regards,

Ilya__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev