BTW a very similar discussion is going in Neutron community right now, please find a thread under the *[openstack-dev] [Neutron] Introducing task oriented workflows* label.
Best regards, Max Lobur, Python Developer, Mirantis, Inc. Mobile: +38 (093) 665 14 28 Skype: max_lobur 38, Lenina ave. Kharkov, Ukraine www.mirantis.com www.mirantis.ru On Wed, May 28, 2014 at 6:56 PM, Maksym Lobur <mlo...@mirantis.com> wrote: > Hi All, > > You've raised a good discussion, something similar already was started > back in february. Could someone please find the long etherpad with > discussion between Deva and Lifeless, as I recall most of the points > mentioned above have a good comments there. > > Up to this point I have the only idea how to elegantly address these > problems. This is a tasks concept and probably a scheduler service, which > not necessarily should be separate from the API at the moment (Finally we > already have a hash ring on the api side which is a kind of scheduler > right?) It was already proposed earlier, but I would like to try to fit all > these issues into this concept. > > >> 1. "Executability" >> We need to make sure that request can be theoretically executed, >> which includes: >> a) Validating request body >> > > We cannot validate everything on the API side, relying on the fact that DB > state is actual is not a good idea, especially under heavy load. > > In tasks concept we could assume that all the requests are executable, and > do not perform any validation in the API thread at all. Instead of this the > API will just create a task and return it's ID to the user. Task scheduler > may perform some minor validations before the task is queued or started for > convenience, but they should be duplicated inside task body because there > is an arbitrary time between queuing up and start ((c)lifeless). I assume > the scheduler will have it's own thread or even process. The user will need > to poke received ID to know the current state of his submission. > > >> b) For each of entities (e.g. nodes) touched, check that they are >> available >> at the moment (at least exist). >> This is arguable, as checking for entity existence requires going to >> DB. > > > Same here, DB round trip is a potential block, therefore this will be done > inside task (after it's queued and started) and will not affect the API. > The user will just observe the task state by poking the API (or using > callback as an option). > > 2. Appropriate state >> For each entity in question, ensure that it's either in a proper state >> or >> moving to a proper state. >> It would help avoid users e.g. setting deploy twice on the same node >> It will still require some kind of NodeInAWrongStateError, but we won't >> necessary need a client retry on this one. >> Allowing the entity to be _moving_ to appropriate state gives us a >> problem: >> Imagine OP1 was running and OP2 got scheduled, hoping that OP1 will come >> to desired state. What if OP1 fails? What if conductor, doing OP1 >> crashes? > > > Let's say OP1 and OP2 are two separate tasks. Each one have the initial > state validation inside it's body. Once OP2 gets its turn it will perform > validation and fail, which looks reasonable to me. > > >> Similar problem with checking node state. >> Imagine we schedule OP2 while we had OP1 - regular checking node state. >> OP1 discovers that node is actually absent and puts it to maintenance >> state. >> What to do with OP2? > > > The task will fail once it get it's turn. > > >> b) Can we make client wait for the results of periodic check? >> That is, wait for OP1 _before scheduling_ OP2? > > > We will just schedule the task and the user will observe its progress, > once OP1 is finished and OP2 started - he will see a fail. > > >> 3. Status feedback >> People would like to know, how things are going with their task. >> What they know is that their request was scheduled. Options: >> a) Poll: return some REQUEST_ID and expect users to poll some endpoint. >> Pros: >> - Should be easy to implement >> Cons: >> - Requires persistent storage for tasks. Does AMQP allow to do this >> kinds >> of queries? If not, we'll need to duplicate tasks in DB. >> - Increased load on API instances and DB >> > > Exactly described the tasks concept :) > > >> b) Callback: take endpoint, call it once task is done/fails. >> Pros: >> - Less load on both client and server >> - Answer exactly when it's ready >> Cons: >> - Will not work for cli and similar >> - If conductor crashes, there will be no callback. > > > Add to Cons: > - Callback is not reliable since it may get lost. > We should have an ability to poke anyway, though I see a great benefit > from implementing a callbacks - to decrease API load. > > >> 4. Debugging consideration >> a) This is an open question: how to debug, if we have a lot of requests >> and something went wrong? >> > > We will be able to see the queue state (btw what about security here, > should the user be able to see all the tasks, or just his ones, or all but > others with hidden details). > > >> b) One more thing to consider: how to make command like `node-show` >> aware of >> scheduled transitioning, so that people don't try operations that are >> doomed to failure. > > > node-show will always show current state of the node, though we may check > if there are any tasks queued or going, which will change the state. If any > - add a notification to the response. > > >> 5. Performance considerations >> a) With async approach, users will be able to schedule nearly unlimited >> number of tasks, thus essentially blocking work of Ironic, without >> any >> signs of the problem (at least for some time). >> I think there are 2 common answers to this problem: >> - Request throttling: disallow user to make too many requests in some >> amount of time. Send them 503 with Retry-After header set. > > > Can this be achieved by some web-server settings? Looks like a typical > problem. > > >> - Queue management: watch queue length, deny new requests if it's too >> large. >> > > Yes, I really like the limited queue size idea. Please see my comments in > the spec. > Also, if we have a tasks and the queue, we could merge similar tasks > > >> b) State framework from (2), if invented, can become a bottleneck as >> well. >> Especially with polling approach. > > > True. > If we have tasks, all the node actions will be done through them. We can > synchronise node state with DB only during the task, and remove periodic > syncs. Off-course someone may go and turn off the node, in this case the > Ironic will lie about the node state until some task is executed on this > node, which may be suitable behaviour. Otherwise rare periodic syncs may > work as well. > > >> 6. Usability considerations >> a) People will be unaware, when and whether their request is going to be >> finished. As there will be tempted to retry, we may get flooded by >> duplicates. I would suggest at least make it possible to request >> canceling >> any task (which will be possible only if it is not started yet, >> obviously). >> > > Since we will have a limited number of kinds of tasks, we could calculate > some estimates basing on previous similar tasks. Looks like an improvement > for a distant future. In the end I wouldn't want Ironic to perform > estimates like windows's copy-paste dialog :) > > Tasks may be easily interrupted while they are in a queue. But if it's > already started - there's a separate dicsussion > https://blueprints.launchpad.net/ironic/+spec/make-tasks-interruptible(I'm > going to port this bp to the specs repo in some time) > > >> b) We should try to avoid scheduling contradictive requests. >> > > A task scheduler responsibility: this is basically a state check before > task is scheduled, and it should be done one more time once the task is > started, as mentioned above. > > >> c) Can we somehow detect duplicated requests and ignore them? >> E.g. we won't want user to make 2-3-4 reboots in a row just because >> the user >> was not patient enough. > > > Queue similar tasks. All the users will be pointed to the similar task > resource, or maybe to a different resources which tied to the same > conductor action. > > >> Best regards, >> Max Lobur, >> Python Developer, Mirantis, Inc. >> Mobile: +38 (093) 665 14 28 >> Skype: max_lobur >> 38, Lenina ave. Kharkov, Ukraine >> www.mirantis.com >> www.mirantis.ru >> >> >> On Wed, May 28, 2014 at 5:10 PM, Lucas Alvares Gomes < >> lucasago...@gmail.com> wrote: >> On Wed, May 28, 2014 at 2:02 PM, Dmitry Tantsur <dtant...@redhat.com> >> wrote: >> > Hi Ironic folks, hi Devananda! >> > >> > I'd like to share with you my thoughts on asynchronous API, which is >> > spec https://review.openstack.org/#/c/94923 >> > First I was planned this as comments to the review, but it proved to be >> > much larger, so I post it for discussion on ML. >> > >> > Here is list of different consideration, I'd like to take into account >> > when prototyping async support, some are reflected in spec already, some >> > are from my and other's comments: >> > >> > 1. "Executability" >> > We need to make sure that request can be theoretically executed, >> > which includes: >> > a) Validating request body >> > b) For each of entities (e.g. nodes) touched, check that they are >> > available >> > at the moment (at least exist). >> > This is arguable, as checking for entity existence requires going to >> > DB. >> >> > >> > 2. Appropriate state >> > For each entity in question, ensure that it's either in a proper state >> > or >> > moving to a proper state. >> > It would help avoid users e.g. setting deploy twice on the same node >> > It will still require some kind of NodeInAWrongStateError, but we won't >> > necessary need a client retry on this one. >> > >> > Allowing the entity to be _moving_ to appropriate state gives us a >> > problem: >> > Imagine OP1 was running and OP2 got scheduled, hoping that OP1 will come >> > to desired state. What if OP1 fails? What if conductor, doing OP1 >> > crashes? >> > That's why we may want to approve only operations on entities that do >> > not >> > undergo state changes. What do you think? >> > >> > Similar problem with checking node state. >> > Imagine we schedule OP2 while we had OP1 - regular checking node state. >> > OP1 discovers that node is actually absent and puts it to maintenance >> > state. >> > What to do with OP2? >> > a) Obvious answer is to fail it >> > b) Can we make client wait for the results of periodic check? >> > That is, wait for OP1 _before scheduling_ OP2? >> > >> > Anyway, this point requires some state framework, that knows about >> > states, >> > transitions, actions and their compatibility with each other. >> For {power, provision} state changes should we queue the requests? We >> may want to only accept 1 request to change the state per time, if a >> second request comes when there's another state change mid-operation >> we may just return 409 (Conflict) to indicate that a state change is >> already in progress. This is similar of what we have today but instead >> of checking the node lock and states on the conductor side the API >> service could do it, since it's on the DB. >> > >> > 3. Status feedback >> > People would like to know, how things are going with their task. >> > What they know is that their request was scheduled. Options: >> > a) Poll: return some REQUEST_ID and expect users to poll some endpoint. >> > Pros: >> > - Should be easy to implement >> > Cons: >> > - Requires persistent storage for tasks. Does AMQP allow to do this >> > kinds >> > of queries? If not, we'll need to duplicate tasks in DB. >> > - Increased load on API instances and DB >> > b) Callback: take endpoint, call it once task is done/fails. >> > Pros: >> > - Less load on both client and server >> > - Answer exactly when it's ready >> > Cons: >> > - Will not work for cli and similar >> > - If conductor crashes, there will be no callback. >> > >> > Seems like we'd want both (a) and (b) to comply with current needs. >> +1, we could allow pooling by default (like checking >> nodes/<uuid>/states to know the current and target state of the node) >> but we may also want to include a callback parameter that users could >> use to input a URL that the conductor will call out as soon as the >> operation is finished. So if the callback URl exists, the conductor >> will submit a POST request to that URL with some data structure >> identifying the operation and the current state. >> > >> > If we have a state framework from (2), we can also add notifications to >> > it. >> > >> > 4. Debugging consideration >> > a) This is an open question: how to debug, if we have a lot of requests >> > and something went wrong? >> > b) One more thing to consider: how to make command like `node-show` >> > aware of >> > scheduled transitioning, so that people don't try operations that are >> > doomed to failure. >> > >> > 5. Performance considerations >> > a) With async approach, users will be able to schedule nearly unlimited >> > number of tasks, thus essentially blocking work of Ironic, without >> > any >> > signs of the problem (at least for some time). >> > I think there are 2 common answers to this problem: >> > - Request throttling: disallow user to make too many requests in some >> > amount of time. Send them 503 with Retry-After header set. >> > - Queue management: watch queue length, deny new requests if it's too >> > large. >> > This means actually getting back error 503 and will require retrying >> > again! >> > At least it will be exceptional case, and won't affect Tempest run... >> > b) State framework from (2), if invented, can become a bottleneck as >> > well. >> > Especially with polling approach. >> > >> > 6. Usability considerations >> > a) People will be unaware, when and whether their request is going to be >> > finished. As there will be tempted to retry, we may get flooded by >> > duplicates. I would suggest at least make it possible to request >> > canceling >> > any task (which will be possible only if it is not started yet, >> > obviously). >> > b) We should try to avoid scheduling contradictive requests. >> > c) Can we somehow detect duplicated requests and ignore them? >> > E.g. we won't want user to make 2-3-4 reboots in a row just because >> > the user >> > was not patient enough. >> > >> > ------ >> > >> > Possible takeaways from this letter: >> > - We'll need at least throttling to avoid DoS >> > - We'll still need handling of 503 error, though it should not happen >> > under >> > normal conditions >> > - Think about state framework that unifies all this complex logic with >> > features: >> > * Track entities, their states and actions on entities >> > * Check whether new action is compatible with states of entities it >> > touches >> > and with other ongoing and scheduled actions on these entities. >> > * Handle notifications for finished and failed actions by providing >> > both >> > pull and push approaches. >> > * Track whether started action is still executed, perform error >> > notification, >> > if not. >> > * HA and high performance >> > - Think about policies for corner cases >> > - Think, how we can make a user aware of what is going on with both >> > request >> > and entity that some requests may touch. Also consider canceling >> > requests. >> > >> > Please let me know, what you think. >> > >> > Dmitry. >> > >> > >> > _______________________________________________ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > + 1 >> > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
