Hi All, You've raised a good discussion, something similar already was started back in february. Could someone please find the long etherpad with discussion between Deva and Lifeless, as I recall most of the points mentioned above have a good comments there.
Up to this point I have the only idea how to elegantly address these problems. This is a tasks concept and probably a scheduler service, which not necessarily should be separate from the API at the moment (Finally we already have a hash ring on the api side which is a kind of scheduler right?) It was already proposed earlier, but I would like to try to fit all these issues into this concept. > 1. "Executability" > We need to make sure that request can be theoretically executed, > which includes: > a) Validating request body > We cannot validate everything on the API side, relying on the fact that DB state is actual is not a good idea, especially under heavy load. In tasks concept we could assume that all the requests are executable, and do not perform any validation in the API thread at all. Instead of this the API will just create a task and return it's ID to the user. Task scheduler may perform some minor validations before the task is queued or started for convenience, but they should be duplicated inside task body because there is an arbitrary time between queuing up and start ((c)lifeless). I assume the scheduler will have it's own thread or even process. The user will need to poke received ID to know the current state of his submission. > b) For each of entities (e.g. nodes) touched, check that they are > available > at the moment (at least exist). > This is arguable, as checking for entity existence requires going to > DB. Same here, DB round trip is a potential block, therefore this will be done inside task (after it's queued and started) and will not affect the API. The user will just observe the task state by poking the API (or using callback as an option). 2. Appropriate state > For each entity in question, ensure that it's either in a proper state > or > moving to a proper state. > It would help avoid users e.g. setting deploy twice on the same node > It will still require some kind of NodeInAWrongStateError, but we won't > necessary need a client retry on this one. > Allowing the entity to be _moving_ to appropriate state gives us a > problem: > Imagine OP1 was running and OP2 got scheduled, hoping that OP1 will come > to desired state. What if OP1 fails? What if conductor, doing OP1 > crashes? Let's say OP1 and OP2 are two separate tasks. Each one have the initial state validation inside it's body. Once OP2 gets its turn it will perform validation and fail, which looks reasonable to me. > Similar problem with checking node state. > Imagine we schedule OP2 while we had OP1 - regular checking node state. > OP1 discovers that node is actually absent and puts it to maintenance > state. > What to do with OP2? The task will fail once it get it's turn. > b) Can we make client wait for the results of periodic check? > That is, wait for OP1 _before scheduling_ OP2? We will just schedule the task and the user will observe its progress, once OP1 is finished and OP2 started - he will see a fail. > 3. Status feedback > People would like to know, how things are going with their task. > What they know is that their request was scheduled. Options: > a) Poll: return some REQUEST_ID and expect users to poll some endpoint. > Pros: > - Should be easy to implement > Cons: > - Requires persistent storage for tasks. Does AMQP allow to do this > kinds > of queries? If not, we'll need to duplicate tasks in DB. > - Increased load on API instances and DB > Exactly described the tasks concept :) > b) Callback: take endpoint, call it once task is done/fails. > Pros: > - Less load on both client and server > - Answer exactly when it's ready > Cons: > - Will not work for cli and similar > - If conductor crashes, there will be no callback. Add to Cons: - Callback is not reliable since it may get lost. We should have an ability to poke anyway, though I see a great benefit from implementing a callbacks - to decrease API load. > 4. Debugging consideration > a) This is an open question: how to debug, if we have a lot of requests > and something went wrong? > We will be able to see the queue state (btw what about security here, should the user be able to see all the tasks, or just his ones, or all but others with hidden details). > b) One more thing to consider: how to make command like `node-show` > aware of > scheduled transitioning, so that people don't try operations that are > doomed to failure. node-show will always show current state of the node, though we may check if there are any tasks queued or going, which will change the state. If any - add a notification to the response. > 5. Performance considerations > a) With async approach, users will be able to schedule nearly unlimited > number of tasks, thus essentially blocking work of Ironic, without > any > signs of the problem (at least for some time). > I think there are 2 common answers to this problem: > - Request throttling: disallow user to make too many requests in some > amount of time. Send them 503 with Retry-After header set. Can this be achieved by some web-server settings? Looks like a typical problem. > - Queue management: watch queue length, deny new requests if it's too > large. > Yes, I really like the limited queue size idea. Please see my comments in the spec. Also, if we have a tasks and the queue, we could merge similar tasks > b) State framework from (2), if invented, can become a bottleneck as > well. > Especially with polling approach. True. If we have tasks, all the node actions will be done through them. We can synchronise node state with DB only during the task, and remove periodic syncs. Off-course someone may go and turn off the node, in this case the Ironic will lie about the node state until some task is executed on this node, which may be suitable behaviour. Otherwise rare periodic syncs may work as well. > 6. Usability considerations > a) People will be unaware, when and whether their request is going to be > finished. As there will be tempted to retry, we may get flooded by > duplicates. I would suggest at least make it possible to request > canceling > any task (which will be possible only if it is not started yet, > obviously). > Since we will have a limited number of kinds of tasks, we could calculate some estimates basing on previous similar tasks. Looks like an improvement for a distant future. In the end I wouldn't want Ironic to perform estimates like windows's copy-paste dialog :) Tasks may be easily interrupted while they are in a queue. But if it's already started - there's a separate dicsussion https://blueprints.launchpad.net/ironic/+spec/make-tasks-interruptible (I'm going to port this bp to the specs repo in some time) > b) We should try to avoid scheduling contradictive requests. > A task scheduler responsibility: this is basically a state check before task is scheduled, and it should be done one more time once the task is started, as mentioned above. > c) Can we somehow detect duplicated requests and ignore them? > E.g. we won't want user to make 2-3-4 reboots in a row just because > the user > was not patient enough. Queue similar tasks. All the users will be pointed to the similar task resource, or maybe to a different resources which tied to the same conductor action. > Best regards, > Max Lobur, > Python Developer, Mirantis, Inc. > Mobile: +38 (093) 665 14 28 > Skype: max_lobur > 38, Lenina ave. Kharkov, Ukraine > www.mirantis.com > www.mirantis.ru > > On Wed, May 28, 2014 at 5:10 PM, Lucas Alvares Gomes < > lucasago...@gmail.com> wrote: > On Wed, May 28, 2014 at 2:02 PM, Dmitry Tantsur <dtant...@redhat.com> > wrote: > > Hi Ironic folks, hi Devananda! > > > > I'd like to share with you my thoughts on asynchronous API, which is > > spec https://review.openstack.org/#/c/94923 > > First I was planned this as comments to the review, but it proved to be > > much larger, so I post it for discussion on ML. > > > > Here is list of different consideration, I'd like to take into account > > when prototyping async support, some are reflected in spec already, some > > are from my and other's comments: > > > > 1. "Executability" > > We need to make sure that request can be theoretically executed, > > which includes: > > a) Validating request body > > b) For each of entities (e.g. nodes) touched, check that they are > > available > > at the moment (at least exist). > > This is arguable, as checking for entity existence requires going to > > DB. > > > > > 2. Appropriate state > > For each entity in question, ensure that it's either in a proper state > > or > > moving to a proper state. > > It would help avoid users e.g. setting deploy twice on the same node > > It will still require some kind of NodeInAWrongStateError, but we won't > > necessary need a client retry on this one. > > > > Allowing the entity to be _moving_ to appropriate state gives us a > > problem: > > Imagine OP1 was running and OP2 got scheduled, hoping that OP1 will come > > to desired state. What if OP1 fails? What if conductor, doing OP1 > > crashes? > > That's why we may want to approve only operations on entities that do > > not > > undergo state changes. What do you think? > > > > Similar problem with checking node state. > > Imagine we schedule OP2 while we had OP1 - regular checking node state. > > OP1 discovers that node is actually absent and puts it to maintenance > > state. > > What to do with OP2? > > a) Obvious answer is to fail it > > b) Can we make client wait for the results of periodic check? > > That is, wait for OP1 _before scheduling_ OP2? > > > > Anyway, this point requires some state framework, that knows about > > states, > > transitions, actions and their compatibility with each other. > For {power, provision} state changes should we queue the requests? We > may want to only accept 1 request to change the state per time, if a > second request comes when there's another state change mid-operation > we may just return 409 (Conflict) to indicate that a state change is > already in progress. This is similar of what we have today but instead > of checking the node lock and states on the conductor side the API > service could do it, since it's on the DB. > > > > 3. Status feedback > > People would like to know, how things are going with their task. > > What they know is that their request was scheduled. Options: > > a) Poll: return some REQUEST_ID and expect users to poll some endpoint. > > Pros: > > - Should be easy to implement > > Cons: > > - Requires persistent storage for tasks. Does AMQP allow to do this > > kinds > > of queries? If not, we'll need to duplicate tasks in DB. > > - Increased load on API instances and DB > > b) Callback: take endpoint, call it once task is done/fails. > > Pros: > > - Less load on both client and server > > - Answer exactly when it's ready > > Cons: > > - Will not work for cli and similar > > - If conductor crashes, there will be no callback. > > > > Seems like we'd want both (a) and (b) to comply with current needs. > +1, we could allow pooling by default (like checking > nodes/<uuid>/states to know the current and target state of the node) > but we may also want to include a callback parameter that users could > use to input a URL that the conductor will call out as soon as the > operation is finished. So if the callback URl exists, the conductor > will submit a POST request to that URL with some data structure > identifying the operation and the current state. > > > > If we have a state framework from (2), we can also add notifications to > > it. > > > > 4. Debugging consideration > > a) This is an open question: how to debug, if we have a lot of requests > > and something went wrong? > > b) One more thing to consider: how to make command like `node-show` > > aware of > > scheduled transitioning, so that people don't try operations that are > > doomed to failure. > > > > 5. Performance considerations > > a) With async approach, users will be able to schedule nearly unlimited > > number of tasks, thus essentially blocking work of Ironic, without > > any > > signs of the problem (at least for some time). > > I think there are 2 common answers to this problem: > > - Request throttling: disallow user to make too many requests in some > > amount of time. Send them 503 with Retry-After header set. > > - Queue management: watch queue length, deny new requests if it's too > > large. > > This means actually getting back error 503 and will require retrying > > again! > > At least it will be exceptional case, and won't affect Tempest run... > > b) State framework from (2), if invented, can become a bottleneck as > > well. > > Especially with polling approach. > > > > 6. Usability considerations > > a) People will be unaware, when and whether their request is going to be > > finished. As there will be tempted to retry, we may get flooded by > > duplicates. I would suggest at least make it possible to request > > canceling > > any task (which will be possible only if it is not started yet, > > obviously). > > b) We should try to avoid scheduling contradictive requests. > > c) Can we somehow detect duplicated requests and ignore them? > > E.g. we won't want user to make 2-3-4 reboots in a row just because > > the user > > was not patient enough. > > > > ------ > > > > Possible takeaways from this letter: > > - We'll need at least throttling to avoid DoS > > - We'll still need handling of 503 error, though it should not happen > > under > > normal conditions > > - Think about state framework that unifies all this complex logic with > > features: > > * Track entities, their states and actions on entities > > * Check whether new action is compatible with states of entities it > > touches > > and with other ongoing and scheduled actions on these entities. > > * Handle notifications for finished and failed actions by providing > > both > > pull and push approaches. > > * Track whether started action is still executed, perform error > > notification, > > if not. > > * HA and high performance > > - Think about policies for corner cases > > - Think, how we can make a user aware of what is going on with both > > request > > and entity that some requests may touch. Also consider canceling > > requests. > > > > Please let me know, what you think. > > > > Dmitry. > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev + 1 >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
