[openstack-dev] [Neutron] Introducing task oriented workflows
As most of you probably know already, this is one of the topics discussed during the Juno summit [1]. I would like to kick off the discussion in order to move towards a concrete design. Preamble: Considering the meat that's already on the plate for Juno, I'm not advocating that whatever comes out of this discussion should be put on the Juno roadmap. However, preparation (or yak shaving) activities that should be identified as pre-requisite might happen during the Juno time frame assuming that they won't interfere with other critical or high priority activities. This is also a very long post; the TL;DR summary is that I would like to explore task-oriented communication with the backend and how it should be reflected in the API - gauging how the community feels about this, and collecting feedback regarding design, constructs, and related tools/techniques/technologies. At the summit a broad range of items were discussed during the session, and most of them have been reported in the etherpad [1]. First, I think it would be good to clarify whether we're advocating a task-based API, a workflow-oriented operation processing, or both. --> About a task-based API In a task-based API, most PUT/POST API operations would return tasks rather than neutron resources, and users of the API will interact directly with tasks. I put an example in [2] to avoid cluttering this post with too much text. As the API operation simply launches a task - the database state won't be updated until the task is completed. Needless to say, this would be a radical change to Neutron's API; it should be carefully evaluated and not considered for the v2 API. Even if it is easily recognisable that this approach has a few benefits, I don't think this will improve usability of the API at all. Indeed this will limit the ability of operating on a resource will a task is in execution on it, and will also require neutron API users to change the paradigm the use to interact with the API; for not mentioning the fact that it would look weird if neutron is the only API endpoint in Openstack operating in this way. For the Neutron API, I think that its operations should still be manipulating the database state, and possibly return immediately after that (*) - a task, or to better say a workflow will then be started, executed asynchronously, and update the resource status on completion. --> On workflow-oriented operations The benefits of it when it comes to easily controlling operations and ensuring consistency in case of failures are obvious. For what is worth, I have been experimenting introducing this kind of capability in the NSX plugin in the past few months. I've been using celery as a task queue, and writing the task management code from scratch - only to realize that the same features I was implementing are already supported by taskflow. I think that all parts of Neutron API can greatly benefit from introducing a flow-based approach. Some examples: - pre/post commit operations in the ML2 plugin can be orchestrated a lot better as a workflow, articulating operations on the various drivers in a graph - operation spanning multiple plugins (eg: add router interface) could be simplified using clearly defined tasks for the L2 and L3 parts - it would be finally possible to properly manage resources' "operational status", as well as knowing whether the actual configuration of the backend matches the database configuration - synchronous plugins might be converted into asynchronous thus improving their API throughput Now, the caveats: - during the sessions it was correctly pointed out that special care is required with multiple producers (ie: api servers) as workflows should be always executed in the correct order - it is probably be advisable to serialize workflows operating on the same resource; this might lead to unexpected situations (potentially to deadlocks) with workflows operating on multiple resources - if the API is asynchronous, and multiple workflows might be queued or in execution at a given time, rolling back the DB operation on failures is probably not advisable (it would not be advisable anyway in any asynchronous framework). If the API instead stays synchronous the revert action for a failed task might also restore the db state for a resource; but I think that keeping the API synchronous missed a bit the point of this whole work - feel free to show your disagreement here! - some neutron workflows are actually initiated by agents; this is the case, for instance, of the workflow for doing initial L2 and security group configuration for a port. - it's going to be a lot of work, and we need to devise a strategy to either roll this changes in the existing plugins or just decide that future v3 plugins will use it. >From the implementation side, I've done a bit of research and task queue like celery only implement half of what is needed; conversely I have not been able to find a workflow manager, at least in the python world, as complete an
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
Hi Salvatore Thank you for your posting this. IMO, this topic shouldn't be limited for Neutron only. Users wants consistent API between OpenStack project, right? In Nova, a server has task_state, so Neutron should do same way. 2014-05-22 15:34 GMT-07:00 Salvatore Orlando : > As most of you probably know already, this is one of the topics discussed > during the Juno summit [1]. > I would like to kick off the discussion in order to move towards a concrete > design. > > Preamble: Considering the meat that's already on the plate for Juno, I'm not > advocating that whatever comes out of this discussion should be put on the > Juno roadmap. However, preparation (or yak shaving) activities that should > be identified as pre-requisite might happen during the Juno time frame > assuming that they won't interfere with other critical or high priority > activities. > This is also a very long post; the TL;DR summary is that I would like to > explore task-oriented communication with the backend and how it should be > reflected in the API - gauging how the community feels about this, and > collecting feedback regarding design, constructs, and related > tools/techniques/technologies. > > At the summit a broad range of items were discussed during the session, and > most of them have been reported in the etherpad [1]. > > First, I think it would be good to clarify whether we're advocating a > task-based API, a workflow-oriented operation processing, or both. > > --> About a task-based API > > In a task-based API, most PUT/POST API operations would return tasks rather > than neutron resources, and users of the API will interact directly with > tasks. > I put an example in [2] to avoid cluttering this post with too much text. > As the API operation simply launches a task - the database state won't be > updated until the task is completed. > > Needless to say, this would be a radical change to Neutron's API; it should > be carefully evaluated and not considered for the v2 API. > Even if it is easily recognisable that this approach has a few benefits, I > don't think this will improve usability of the API at all. Indeed this will > limit the ability of operating on a resource will a task is in execution on > it, and will also require neutron API users to change the paradigm the use > to interact with the API; for not mentioning the fact that it would look > weird if neutron is the only API endpoint in Openstack operating in this > way. > For the Neutron API, I think that its operations should still be > manipulating the database state, and possibly return immediately after that > (*) - a task, or to better say a workflow will then be started, executed > asynchronously, and update the resource status on completion. > > --> On workflow-oriented operations > > The benefits of it when it comes to easily controlling operations and > ensuring consistency in case of failures are obvious. For what is worth, I > have been experimenting introducing this kind of capability in the NSX > plugin in the past few months. I've been using celery as a task queue, and > writing the task management code from scratch - only to realize that the > same features I was implementing are already supported by taskflow. > > I think that all parts of Neutron API can greatly benefit from introducing a > flow-based approach. > Some examples: > - pre/post commit operations in the ML2 plugin can be orchestrated a lot > better as a workflow, articulating operations on the various drivers in a > graph > - operation spanning multiple plugins (eg: add router interface) could be > simplified using clearly defined tasks for the L2 and L3 parts > - it would be finally possible to properly manage resources' "operational > status", as well as knowing whether the actual configuration of the backend > matches the database configuration > - synchronous plugins might be converted into asynchronous thus improving > their API throughput > > Now, the caveats: > - during the sessions it was correctly pointed out that special care is > required with multiple producers (ie: api servers) as workflows should be > always executed in the correct order > - it is probably be advisable to serialize workflows operating on the same > resource; this might lead to unexpected situations (potentially to > deadlocks) with workflows operating on multiple resources > - if the API is asynchronous, and multiple workflows might be queued or in > execution at a given time, rolling back the DB operation on failures is > probably not advisable (it would not be advisable anyway in any asynchronous > framework). If the API instead stays synchronous the revert action for a > failed task might also restore the db state for a resource; but I think that > keeping the API synchronous missed a bit the point of this whole work - feel > free to show your disagreement here! > - some neutron workflows are actually initiated by agents; this is the case, > for instance, of the workflow for doing initial L2 and s
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
Nachi, I will be glad if the solution was as easy as sticking a task_state attribute to a resource! I'm afraid however that would be only the tip of the iceberg, or the icing of the cake, if you want. However, I agree with you that consistency across Openstack APIs is very important; whether this is a cross project discussion is instead debatable; my feeling here is that taskflow is the cross-project piece of the architecture, and every project then might have a different strategy for integrating it - as long as it does not result in inconsistent APIs exposed to customers! It is something that obviously will be considered when designing how to represent whether a DB resource is in sync with its actual configuration on the backend. I think this is something which might happen regardless of whether it will be also agreed to let API consumers access task execution information using the API. Salvatore On 23 May 2014 01:16, Nachi Ueno wrote: > Hi Salvatore > > Thank you for your posting this. > > IMO, this topic shouldn't be limited for Neutron only. > Users wants consistent API between OpenStack project, right? > > In Nova, a server has task_state, so Neutron should do same way. > > > > 2014-05-22 15:34 GMT-07:00 Salvatore Orlando : > > As most of you probably know already, this is one of the topics discussed > > during the Juno summit [1]. > > I would like to kick off the discussion in order to move towards a > concrete > > design. > > > > Preamble: Considering the meat that's already on the plate for Juno, I'm > not > > advocating that whatever comes out of this discussion should be put on > the > > Juno roadmap. However, preparation (or yak shaving) activities that > should > > be identified as pre-requisite might happen during the Juno time frame > > assuming that they won't interfere with other critical or high priority > > activities. > > This is also a very long post; the TL;DR summary is that I would like to > > explore task-oriented communication with the backend and how it should be > > reflected in the API - gauging how the community feels about this, and > > collecting feedback regarding design, constructs, and related > > tools/techniques/technologies. > > > > At the summit a broad range of items were discussed during the session, > and > > most of them have been reported in the etherpad [1]. > > > > First, I think it would be good to clarify whether we're advocating a > > task-based API, a workflow-oriented operation processing, or both. > > > > --> About a task-based API > > > > In a task-based API, most PUT/POST API operations would return tasks > rather > > than neutron resources, and users of the API will interact directly with > > tasks. > > I put an example in [2] to avoid cluttering this post with too much text. > > As the API operation simply launches a task - the database state won't be > > updated until the task is completed. > > > > Needless to say, this would be a radical change to Neutron's API; it > should > > be carefully evaluated and not considered for the v2 API. > > Even if it is easily recognisable that this approach has a few benefits, > I > > don't think this will improve usability of the API at all. Indeed this > will > > limit the ability of operating on a resource will a task is in execution > on > > it, and will also require neutron API users to change the paradigm the > use > > to interact with the API; for not mentioning the fact that it would look > > weird if neutron is the only API endpoint in Openstack operating in this > > way. > > For the Neutron API, I think that its operations should still be > > manipulating the database state, and possibly return immediately after > that > > (*) - a task, or to better say a workflow will then be started, executed > > asynchronously, and update the resource status on completion. > > > > --> On workflow-oriented operations > > > > The benefits of it when it comes to easily controlling operations and > > ensuring consistency in case of failures are obvious. For what is worth, > I > > have been experimenting introducing this kind of capability in the NSX > > plugin in the past few months. I've been using celery as a task queue, > and > > writing the task management code from scratch - only to realize that the > > same features I was implementing are already supported by taskflow. > > > > I think that all parts of Neutron API can greatly benefit from > introducing a > > flow-based approach. > > Some examples: > > - pre/post commit operations in the ML2 plugin can be orchestrated a lot > > better as a workflow, articulating operations on the various drivers in a > > graph > > - operation spanning multiple plugins (eg: add router interface) could be > > simplified using clearly defined tasks for the L2 and L3 parts > > - it would be finally possible to properly manage resources' "operational > > status", as well as knowing whether the actual configuration of the > backend > > matches the database configuration > > - synchro
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
Hello, I think the Mistral Project[1] aims the same goal, isn't it? Regards, jaume [1]: https://wiki.openstack.org/wiki/Mistral On 23 May 2014 09:28, Salvatore Orlando wrote: > Nachi, > > I will be glad if the solution was as easy as sticking a task_state > attribute to a resource! I'm afraid however that would be only the tip of > the iceberg, or the icing of the cake, if you want. > However, I agree with you that consistency across Openstack APIs is very > important; whether this is a cross project discussion is instead debatable; > my feeling here is that taskflow is the cross-project piece of the > architecture, and every project then might have a different strategy for > integrating it - as long as it does not result in inconsistent APIs exposed > to customers! > > It is something that obviously will be considered when designing how to > represent whether a DB resource is in sync with its actual configuration on > the backend. > I think this is something which might happen regardless of whether it will > be also agreed to let API consumers access task execution information using > the API. > > Salvatore > > > > > On 23 May 2014 01:16, Nachi Ueno wrote: > >> Hi Salvatore >> >> Thank you for your posting this. >> >> IMO, this topic shouldn't be limited for Neutron only. >> Users wants consistent API between OpenStack project, right? >> >> In Nova, a server has task_state, so Neutron should do same way. >> >> >> >> 2014-05-22 15:34 GMT-07:00 Salvatore Orlando : >> > As most of you probably know already, this is one of the topics >> discussed >> > during the Juno summit [1]. >> > I would like to kick off the discussion in order to move towards a >> concrete >> > design. >> > >> > Preamble: Considering the meat that's already on the plate for Juno, >> I'm not >> > advocating that whatever comes out of this discussion should be put on >> the >> > Juno roadmap. However, preparation (or yak shaving) activities that >> should >> > be identified as pre-requisite might happen during the Juno time frame >> > assuming that they won't interfere with other critical or high priority >> > activities. >> > This is also a very long post; the TL;DR summary is that I would like to >> > explore task-oriented communication with the backend and how it should >> be >> > reflected in the API - gauging how the community feels about this, and >> > collecting feedback regarding design, constructs, and related >> > tools/techniques/technologies. >> > >> > At the summit a broad range of items were discussed during the session, >> and >> > most of them have been reported in the etherpad [1]. >> > >> > First, I think it would be good to clarify whether we're advocating a >> > task-based API, a workflow-oriented operation processing, or both. >> > >> > --> About a task-based API >> > >> > In a task-based API, most PUT/POST API operations would return tasks >> rather >> > than neutron resources, and users of the API will interact directly with >> > tasks. >> > I put an example in [2] to avoid cluttering this post with too much >> text. >> > As the API operation simply launches a task - the database state won't >> be >> > updated until the task is completed. >> > >> > Needless to say, this would be a radical change to Neutron's API; it >> should >> > be carefully evaluated and not considered for the v2 API. >> > Even if it is easily recognisable that this approach has a few >> benefits, I >> > don't think this will improve usability of the API at all. Indeed this >> will >> > limit the ability of operating on a resource will a task is in >> execution on >> > it, and will also require neutron API users to change the paradigm the >> use >> > to interact with the API; for not mentioning the fact that it would look >> > weird if neutron is the only API endpoint in Openstack operating in this >> > way. >> > For the Neutron API, I think that its operations should still be >> > manipulating the database state, and possibly return immediately after >> that >> > (*) - a task, or to better say a workflow will then be started, executed >> > asynchronously, and update the resource status on completion. >> > >> > --> On workflow-oriented operations >> > >> > The benefits of it when it comes to easily controlling operations and >> > ensuring consistency in case of failures are obvious. For what is >> worth, I >> > have been experimenting introducing this kind of capability in the NSX >> > plugin in the past few months. I've been using celery as a task queue, >> and >> > writing the task management code from scratch - only to realize that the >> > same features I was implementing are already supported by taskflow. >> > >> > I think that all parts of Neutron API can greatly benefit from >> introducing a >> > flow-based approach. >> > Some examples: >> > - pre/post commit operations in the ML2 plugin can be orchestrated a lot >> > better as a workflow, articulating operations on the various drivers in >> a >> > graph >> > - operation spanning m
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
I think Cinder has some of the same sauce ? https://review.openstack.org/#/c/94742/ https://review.openstack.org/#/c/95037/ 2014-05-23 10:57 GMT+02:00 Jaume Devesa : > Hello, > > I think the Mistral Project[1] aims the same goal, isn't it? > > Regards, > jaume > > [1]: https://wiki.openstack.org/wiki/Mistral > > > On 23 May 2014 09:28, Salvatore Orlando wrote: > >> Nachi, >> >> I will be glad if the solution was as easy as sticking a task_state >> attribute to a resource! I'm afraid however that would be only the tip of >> the iceberg, or the icing of the cake, if you want. >> However, I agree with you that consistency across Openstack APIs is very >> important; whether this is a cross project discussion is instead debatable; >> my feeling here is that taskflow is the cross-project piece of the >> architecture, and every project then might have a different strategy for >> integrating it - as long as it does not result in inconsistent APIs exposed >> to customers! >> >> It is something that obviously will be considered when designing how to >> represent whether a DB resource is in sync with its actual configuration on >> the backend. >> I think this is something which might happen regardless of whether it >> will be also agreed to let API consumers access task execution information >> using the API. >> >> Salvatore >> >> >> >> >> On 23 May 2014 01:16, Nachi Ueno wrote: >> >>> Hi Salvatore >>> >>> Thank you for your posting this. >>> >>> IMO, this topic shouldn't be limited for Neutron only. >>> Users wants consistent API between OpenStack project, right? >>> >>> In Nova, a server has task_state, so Neutron should do same way. >>> >>> >>> >>> 2014-05-22 15:34 GMT-07:00 Salvatore Orlando : >>> > As most of you probably know already, this is one of the topics >>> discussed >>> > during the Juno summit [1]. >>> > I would like to kick off the discussion in order to move towards a >>> concrete >>> > design. >>> > >>> > Preamble: Considering the meat that's already on the plate for Juno, >>> I'm not >>> > advocating that whatever comes out of this discussion should be put on >>> the >>> > Juno roadmap. However, preparation (or yak shaving) activities that >>> should >>> > be identified as pre-requisite might happen during the Juno time frame >>> > assuming that they won't interfere with other critical or high priority >>> > activities. >>> > This is also a very long post; the TL;DR summary is that I would like >>> to >>> > explore task-oriented communication with the backend and how it should >>> be >>> > reflected in the API - gauging how the community feels about this, and >>> > collecting feedback regarding design, constructs, and related >>> > tools/techniques/technologies. >>> > >>> > At the summit a broad range of items were discussed during the >>> session, and >>> > most of them have been reported in the etherpad [1]. >>> > >>> > First, I think it would be good to clarify whether we're advocating a >>> > task-based API, a workflow-oriented operation processing, or both. >>> > >>> > --> About a task-based API >>> > >>> > In a task-based API, most PUT/POST API operations would return tasks >>> rather >>> > than neutron resources, and users of the API will interact directly >>> with >>> > tasks. >>> > I put an example in [2] to avoid cluttering this post with too much >>> text. >>> > As the API operation simply launches a task - the database state won't >>> be >>> > updated until the task is completed. >>> > >>> > Needless to say, this would be a radical change to Neutron's API; it >>> should >>> > be carefully evaluated and not considered for the v2 API. >>> > Even if it is easily recognisable that this approach has a few >>> benefits, I >>> > don't think this will improve usability of the API at all. Indeed this >>> will >>> > limit the ability of operating on a resource will a task is in >>> execution on >>> > it, and will also require neutron API users to change the paradigm the >>> use >>> > to interact with the API; for not mentioning the fact that it would >>> look >>> > weird if neutron is the only API endpoint in Openstack operating in >>> this >>> > way. >>> > For the Neutron API, I think that its operations should still be >>> > manipulating the database state, and possibly return immediately after >>> that >>> > (*) - a task, or to better say a workflow will then be started, >>> executed >>> > asynchronously, and update the resource status on completion. >>> > >>> > --> On workflow-oriented operations >>> > >>> > The benefits of it when it comes to easily controlling operations and >>> > ensuring consistency in case of failures are obvious. For what is >>> worth, I >>> > have been experimenting introducing this kind of capability in the NSX >>> > plugin in the past few months. I've been using celery as a task queue, >>> and >>> > writing the task management code from scratch - only to realize that >>> the >>> > same features I was implementing are already supported by taskflow. >>> > >>> >
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
While Mistral is a service with its own REST API endpoint, taskflow is a library (shoot me if I'm wrong here). Also, mistral appears, in my opinion, to satisfy a set of use cases aimed at cloud operators rather than for building tasks within an application. These are the reasons for which I did not consider mistral. I don't think it a good idea to rely on a third party service for managing the operations needed to complete a neutron API. Regarding Cinder, the answer is yes; the project is trying to do similar transformations to the API as the ones neutron is aiming to achieve. Salvatore On 23 May 2014 14:26, Endre Karlson wrote: > I think Cinder has some of the same sauce ? > > https://review.openstack.org/#/c/94742/ > https://review.openstack.org/#/c/95037/ > > > > 2014-05-23 10:57 GMT+02:00 Jaume Devesa : > > Hello, >> >> I think the Mistral Project[1] aims the same goal, isn't it? >> >> Regards, >> jaume >> >> [1]: https://wiki.openstack.org/wiki/Mistral >> >> >> On 23 May 2014 09:28, Salvatore Orlando wrote: >> >>> Nachi, >>> >>> I will be glad if the solution was as easy as sticking a task_state >>> attribute to a resource! I'm afraid however that would be only the tip of >>> the iceberg, or the icing of the cake, if you want. >>> However, I agree with you that consistency across Openstack APIs is very >>> important; whether this is a cross project discussion is instead debatable; >>> my feeling here is that taskflow is the cross-project piece of the >>> architecture, and every project then might have a different strategy for >>> integrating it - as long as it does not result in inconsistent APIs exposed >>> to customers! >>> >>> It is something that obviously will be considered when designing how to >>> represent whether a DB resource is in sync with its actual configuration on >>> the backend. >>> I think this is something which might happen regardless of whether it >>> will be also agreed to let API consumers access task execution information >>> using the API. >>> >>> Salvatore >>> >>> >>> >>> >>> On 23 May 2014 01:16, Nachi Ueno wrote: >>> Hi Salvatore Thank you for your posting this. IMO, this topic shouldn't be limited for Neutron only. Users wants consistent API between OpenStack project, right? In Nova, a server has task_state, so Neutron should do same way. 2014-05-22 15:34 GMT-07:00 Salvatore Orlando : > As most of you probably know already, this is one of the topics discussed > during the Juno summit [1]. > I would like to kick off the discussion in order to move towards a concrete > design. > > Preamble: Considering the meat that's already on the plate for Juno, I'm not > advocating that whatever comes out of this discussion should be put on the > Juno roadmap. However, preparation (or yak shaving) activities that should > be identified as pre-requisite might happen during the Juno time frame > assuming that they won't interfere with other critical or high priority > activities. > This is also a very long post; the TL;DR summary is that I would like to > explore task-oriented communication with the backend and how it should be > reflected in the API - gauging how the community feels about this, and > collecting feedback regarding design, constructs, and related > tools/techniques/technologies. > > At the summit a broad range of items were discussed during the session, and > most of them have been reported in the etherpad [1]. > > First, I think it would be good to clarify whether we're advocating a > task-based API, a workflow-oriented operation processing, or both. > > --> About a task-based API > > In a task-based API, most PUT/POST API operations would return tasks rather > than neutron resources, and users of the API will interact directly with > tasks. > I put an example in [2] to avoid cluttering this post with too much text. > As the API operation simply launches a task - the database state won't be > updated until the task is completed. > > Needless to say, this would be a radical change to Neutron's API; it should > be carefully evaluated and not considered for the v2 API. > Even if it is easily recognisable that this approach has a few benefits, I > don't think this will improve usability of the API at all. Indeed this will > limit the ability of operating on a resource will a task is in execution on > it, and will also require neutron API users to change the paradigm the use > to interact with the API; for not mentioning the fact that it would look > weird if neutron is the only API endpoint in Openstack operating in this > way. > For the Neutron API, I think that its operations should still be > man
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
On 23/05/14 01:34, Salvatore Orlando wrote: > As most of you probably know already, this is one of the topics discussed > during the Juno summit [1]. > I would like to kick off the discussion in order to move towards a concrete > design. > > Preamble: Considering the meat that's already on the plate for Juno, I'm > not advocating that whatever comes out of this discussion should be put on > the Juno roadmap. However, preparation (or yak shaving) activities that > should be identified as pre-requisite might happen during the Juno time > frame assuming that they won't interfere with other critical or high > priority activities. > This is also a very long post; the TL;DR summary is that I would like to > explore task-oriented communication with the backend and how it should be > reflected in the API - gauging how the community feels about this, and > collecting feedback regarding design, constructs, and related > tools/techniques/technologies. fwiw, I was previously involved with the DMTF CIMI spec - the 'Job' resource defined there seems match the notion of 'task' as below [1]. The spec has a notion of nesting Jobs but any scheduling (as discussed in the etherpad you link below) was under discussion/tbd. > > At the summit a broad range of items were discussed during the session, and > most of them have been reported in the etherpad [1]. fwiw, there is mention in that etherpad about deferring a task/api call (so once you have a notion of 'Job' you can then schedule its execution). IMO this is an aside/added/unnecessary complication to what is potentially already a huge change. > > First, I think it would be good to clarify whether we're advocating a > task-based API, a workflow-oriented operation processing, or both. > > --> About a task-based API > > In a task-based API, most PUT/POST API operations would return tasks rather > than neutron resources, and users of the API will interact directly with > tasks. > I put an example in [2] to avoid cluttering this post with too much text. > As the API operation simply launches a task - the database state won't be > updated until the task is completed. (as an aside, in the example, another issue is what to return for GET /actual_resource whilst the task is still pending, you kind of cover that with the expanded status semantics discussed later): > > Needless to say, this would be a radical change to Neutron's API; it should > be carefully evaluated and not considered for the v2 API. > Even if it is easily recognisable that this approach has a few benefits, I > don't think this will improve usability of the API at all. Indeed this will > limit the ability of operating on a resource will a task is in execution on > it, and will also require neutron API users to change the paradigm the use > to interact with the API; for not mentioning the fact that it would look > weird if neutron is the only API endpoint in Openstack operating in this > way. +1 - I don't like this; a client will always get back a 'task' with a reference to the actual resource being acted upon. The client would then need to make a further API call to retrieve the (e.g.) created resource. BUT, how can we enable a task API without this model? Could we use an extension header to communicate the task URI? So API would remain unchanged, returning all the usual status but with the task URI included where necessary > For the Neutron API, I think that its operations should still be > manipulating the database state, and possibly return immediately after that > (*) - a task, or to better say a workflow will then be started, executed > asynchronously, and update the resource status on completion. you mean e.g. for create, client gets back 200 OK with the created resource inline, OR 202 Accepted for async tasks (so in the latter case resource state will be 'pending' until its completed, like server create in nova for example?). > > --> On workflow-oriented operations please excuse my ignorance, but I am not entirely sure what the distinction between 'task' and 'workflow' is (I wasn't at summit/discussion if that is some excuse). Is a workflow a directed graph of tasks? > > The benefits of it when it comes to easily controlling operations and > ensuring consistency in case of failures are obvious. For what is worth, I > have been experimenting introducing this kind of capability in the NSX > plugin in the past few months. I've been using celery as a task queue, and > writing the task management code from scratch - only to realize that the > same features I was implementing are already supported by taskflow. > > I think that all parts of Neutron API can greatly benefit from introducing > a flow-based approach. > Some examples: > - pre/post commit operations in the ML2 plugin can be orchestrated a lot > better as a workflow, articulating operations on the various drivers in a > graph > - operation spanning multiple plugins (eg: add router interface) could be > simplified using clearly defined t
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
Hi, Salvatore I think neutron needs the task management too. IMO, the problem of neutron resource status should be discussed individually. Task management enable neutron to roll back API operation and delete trash of resource, try API operation again in one API process. Of course, we can use task to correct inconsistency between neutron DB(resource status) and actual resource configuration. But, we should add resource status management to some resources before task. For example, LBaaS has resource status management[1]. Neutron router, port don't mange status is basic problem. > For instance a port is "UP" if it's been wired by the OVS agent; it often > does not tell us whether the actual resource configuration is exactly the > desired one in the database. For instance, if the ovs agent fails to apply > security groups to a port, the port stays "ACTIVE" and the user might never > know there was an error and the actual state diverged from the desired one. So, we should solve this problem by resource status management such LBaaS rather than task. I don't deny task, but we need to discuss for task long term, I hope the status management will be modified right away. [1] https://wiki.openstack.org/wiki/Neutron/LBaaS/API_1.0#Synchronous_versus_Asynchronous_Plugin_Behavior thanks, Hirofumi - Hirofumi Ichihara NTT Software Innovation Center Tel:+81-422-59-2843 Fax:+81-422-59-2699 Email:ichihara.hirof...@lab.ntt.co.jp - On 2014/05/23, at 7:34, Salvatore Orlando wrote: > As most of you probably know already, this is one of the topics discussed > during the Juno summit [1]. > I would like to kick off the discussion in order to move towards a concrete > design. > > Preamble: Considering the meat that's already on the plate for Juno, I'm not > advocating that whatever comes out of this discussion should be put on the > Juno roadmap. However, preparation (or yak shaving) activities that should be > identified as pre-requisite might happen during the Juno time frame assuming > that they won't interfere with other critical or high priority activities. > This is also a very long post; the TL;DR summary is that I would like to > explore task-oriented communication with the backend and how it should be > reflected in the API - gauging how the community feels about this, and > collecting feedback regarding design, constructs, and related > tools/techniques/technologies. > > At the summit a broad range of items were discussed during the session, and > most of them have been reported in the etherpad [1]. > > First, I think it would be good to clarify whether we're advocating a > task-based API, a workflow-oriented operation processing, or both. > > --> About a task-based API > > In a task-based API, most PUT/POST API operations would return tasks rather > than neutron resources, and users of the API will interact directly with > tasks. > I put an example in [2] to avoid cluttering this post with too much text. > As the API operation simply launches a task - the database state won't be > updated until the task is completed. > > Needless to say, this would be a radical change to Neutron's API; it should > be carefully evaluated and not considered for the v2 API. > Even if it is easily recognisable that this approach has a few benefits, I > don't think this will improve usability of the API at all. Indeed this will > limit the ability of operating on a resource will a task is in execution on > it, and will also require neutron API users to change the paradigm the use to > interact with the API; for not mentioning the fact that it would look weird > if neutron is the only API endpoint in Openstack operating in this way. > For the Neutron API, I think that its operations should still be manipulating > the database state, and possibly return immediately after that (*) - a task, > or to better say a workflow will then be started, executed asynchronously, > and update the resource status on completion. > > --> On workflow-oriented operations > > The benefits of it when it comes to easily controlling operations and > ensuring consistency in case of failures are obvious. For what is worth, I > have been experimenting introducing this kind of capability in the NSX plugin > in the past few months. I've been using celery as a task queue, and writing > the task management code from scratch - only to realize that the same > features I was implementing are already supported by taskflow. > > I think that all parts of Neutron API can greatly benefit from introducing a > flow-based approach. > Some examples: > - pre/post commit operations in the ML2 plugin can be orchestrated a lot > better as a workflow, articulating operations on the various drivers in a > graph > - operation spanning multiple plugins (eg: add router interface) could be > simplified using clearly defined tasks for the L2 and L3 pa
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
Hi Hirofumi, I reckon this has been immediately recognised as a long term effort. However, I just want to clarify that by "long term" I don't mean pushing it back until we get to the next release cycle and we realize we are in the same place where we are today! It is totally correct that most Neutron resources have a sloppy status management. Mostly because, as already pointed out, the 'status' for most resource was conceived to be a 'network fabric' status rather than a resource synchronisation status. As it emerged from previous posts in this thread, I reckon we have three choices: 1) Add a new attribute for describing "configuration" state. For instance this will have values such as PENDING_UPDATE, PENDING_DELETE, IN_SYNC, OUT_OF_SYNC, etc. 2) Merge status and configuration statuses in a single attribute. This will probably result simpler from a client perspective, but there are open questions such as whether a resource for which a task is in progress and is down should be reported as 'down' or 'pending_updage'. 3) Not use any new flags, and use tasks to describe whether there are operations in progress on a resource. The status attribute will describe exclusively the 'fabric' status of a resources; however tasks will be exposed through the API - and a resource in sync will be a resource with no PENDING or FAILED task active on it. The above are just options at the moment; I tend to lean toward the latter, but it would be great to have your feedback. Salvatore On 28 May 2014 11:20, Hirofumi Ichihara wrote: > Hi, Salvatore > > I think neutron needs the task management too. > > IMO, the problem of neutron resource status should be discussed > individually. > Task management enable neutron to roll back API operation and delete trash > of resource, try API operation again in one API process. > Of course, we can use task to correct inconsistency between neutron > DB(resource status) and actual resource configuration. > But, we should add resource status management to some resources before > task. > For example, LBaaS has resource status management[1]. > Neutron router, port don't mange status is basic problem. > > For instance a port is "UP" if it's been wired by the OVS agent; it often > does not tell us whether the actual resource configuration is exactly the > desired one in the database. For instance, if the ovs agent fails to apply > security groups to a port, the port stays "ACTIVE" and the user might never > know there was an error and the actual state diverged from the desired one. > > So, we should solve this problem by resource status management such LBaaS > rather than task. > > I don't deny task, but we need to discuss for task long term, I hope the > status management will be modified right away. > > [1] > https://wiki.openstack.org/wiki/Neutron/LBaaS/API_1.0#Synchronous_versus_Asynchronous_Plugin_Behavior > > thanks, > Hirofumi > > - > Hirofumi Ichihara > NTT Software Innovation Center > Tel:+81-422-59-2843 Fax:+81-422-59-2699 > Email:ichihara.hirof...@lab.ntt.co.jp > - > > > On 2014/05/23, at 7:34, Salvatore Orlando wrote: > > As most of you probably know already, this is one of the topics discussed > during the Juno summit [1]. > I would like to kick off the discussion in order to move towards a > concrete design. > > Preamble: Considering the meat that's already on the plate for Juno, I'm > not advocating that whatever comes out of this discussion should be put on > the Juno roadmap. However, preparation (or yak shaving) activities that > should be identified as pre-requisite might happen during the Juno time > frame assuming that they won't interfere with other critical or high > priority activities. > This is also a very long post; the TL;DR summary is that I would like to > explore task-oriented communication with the backend and how it should be > reflected in the API - gauging how the community feels about this, and > collecting feedback regarding design, constructs, and related > tools/techniques/technologies. > > At the summit a broad range of items were discussed during the session, > and most of them have been reported in the etherpad [1]. > > First, I think it would be good to clarify whether we're advocating a > task-based API, a workflow-oriented operation processing, or both. > > --> About a task-based API > > In a task-based API, most PUT/POST API operations would return tasks > rather than neutron resources, and users of the API will interact directly > with tasks. > I put an example in [2] to avoid cluttering this post with too much text. > As the API operation simply launches a task - the database state won't be > updated until the task is completed. > > Needless to say, this would be a radical change to Neutron's API; it > should be carefully evaluated and not considered for the v2 API. > Even if it is easily recognisable that this approach has a few benefits, I > don't think
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
Hi, Salvatore > It is totally correct that most Neutron resources have a sloppy status > management. Mostly because, as already pointed out, the 'status' for most > resource was conceived to be a 'network fabric' status rather than a resource > synchronisation status. Exactly, I reckon that neutron needs resource synchronization status. > As it emerged from previous posts in this thread, I reckon we have three > choices: > 1) Add a new attribute for describing "configuration" state. For instance > this will have values such as PENDING_UPDATE, PENDING_DELETE, IN_SYNC, > OUT_OF_SYNC, etc. > 2) Merge status and configuration statuses in a single attribute. This will > probably result simpler from a client perspective, but there are open > questions such as whether a resource for which a task is in progress and is > down should be reported as 'down' or 'pending_updage'. > 3) Not use any new flags, and use tasks to describe whether there are > operations in progress on a resource. > The status attribute will describe exclusively the 'fabric' status of a > resources; however tasks will be exposed through the API - and a resource in > sync will be a resource with no PENDING or FAILED task active on it. Good suggestions. I reckon that choice (3) is discussion about new API and choice (1) (2) are discussion about current API. It is not good the problem of current API continues remaining in the future. So they should be discussed individually and improve the fabric status by (1) or (2). When (3) will be achieved, if neutron has same fabric status problem, users may be confused about the difference between resource status and task status. Additionally, to be exact, the task show not resource status but API process status. I reckon we should improve the fabric status, then add task to neutron. Also, I think (2) is good. Because there is performance of LBaaS model. thanks, Hirofumi - 市原 裕史 (Ichihara Hirofumi) NTTソフトウェアイノベーションセンタ Tel:0422-59-2843 Fax:0422-59-2699 Email:ichihara.hirof...@lab.ntt.co.jp - On 2014/05/30, at 17:57, Salvatore Orlando wrote: > Hi Hirofumi, > > I reckon this has been immediately recognised as a long term effort. > However, I just want to clarify that by "long term" I don't mean pushing it > back until we get to the next release cycle and we realize we are in the same > place where we are today! > > It is totally correct that most Neutron resources have a sloppy status > management. Mostly because, as already pointed out, the 'status' for most > resource was conceived to be a 'network fabric' status rather than a resource > synchronisation status. > > As it emerged from previous posts in this thread, I reckon we have three > choices: > 1) Add a new attribute for describing "configuration" state. For instance > this will have values such as PENDING_UPDATE, PENDING_DELETE, IN_SYNC, > OUT_OF_SYNC, etc. > 2) Merge status and configuration statuses in a single attribute. This will > probably result simpler from a client perspective, but there are open > questions such as whether a resource for which a task is in progress and is > down should be reported as 'down' or 'pending_updage'. > 3) Not use any new flags, and use tasks to describe whether there are > operations in progress on a resource. > The status attribute will describe exclusively the 'fabric' status of a > resources; however tasks will be exposed through the API - and a resource in > sync will be a resource with no PENDING or FAILED task active on it. > > The above are just options at the moment; I tend to lean toward the latter, > but it would be great to have your feedback. > > Salvatore > > > > On 28 May 2014 11:20, Hirofumi Ichihara > wrote: > Hi, Salvatore > > I think neutron needs the task management too. > > IMO, the problem of neutron resource status should be discussed individually. > Task management enable neutron to roll back API operation and delete trash of > resource, try API operation again in one API process. > Of course, we can use task to correct inconsistency between neutron > DB(resource status) and actual resource configuration. > But, we should add resource status management to some resources before task. > For example, LBaaS has resource status management[1]. > Neutron router, port don't mange status is basic problem. > >> For instance a port is "UP" if it's been wired by the OVS agent; it often >> does not tell us whether the actual resource configuration is exactly the >> desired one in the database. For instance, if the ovs agent fails to apply >> security groups to a port, the port stays "ACTIVE" and the user might never >> know there was an error and the actual state diverged from the desired one. > So, we should solve this problem by resource status management such LBaaS > rather than task. > > I don't deny task, but we need to discuss for task long term, I hop
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
On 23 May 2014 10:34, Salvatore Orlando wrote: > As most of you probably know already, this is one of the topics discussed > during the Juno summit [1]. > I would like to kick off the discussion in order to move towards a concrete > design. > > Preamble: Considering the meat that's already on the plate for Juno, I'm not > advocating that whatever comes out of this discussion should be put on the > Juno roadmap. However, preparation (or yak shaving) activities that should > be identified as pre-requisite might happen during the Juno time frame > assuming that they won't interfere with other critical or high priority > activities. > This is also a very long post; the TL;DR summary is that I would like to > explore task-oriented communication with the backend and how it should be > reflected in the API - gauging how the community feels about this, and > collecting feedback regarding design, constructs, and related > tools/techniques/technologies. Hi, thanks for writing this up. A few thoughts: - if there can be only one task on a resource at a time, you're essentially forcing all other clients to poll for task completion before coming back to do *their* change. Its kindof a pathological edge case of no in-flight-conflicts :). - Please please please don't embed polling into the design - use webhooks or something similar so that each client (be that Nova, Ironic, Horizon or what-have-you - can get a push response when the thing they want to happen has happened). - I'd think very very carefully about whether you're actually modelling /tasks/ or whether tasks are the implementation and really the core issue is modelling the desired vs obtained resource state - Ironic has a debate going on right now about very much the same problem - the latency involved in some API tasks, and whether the API should return when complete, or when the work his guaranteed to start, or even immediately and maybe the work isn't guaranteed to start. My feeling is that we need to balance ease and correctness of implementation, ease (and efficiency/correctness) of use, and robustness - an entirely non-blocking API might end up being the polling nightmare of nightmares if not done carefully, for instance. -Rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
When considering user interfaces, the choice of task and/or status reporting methods can have a big impact on the ability to communicate with the user. In particular, status properties (e.g. in the manner of compute V2 servers) prevent user interfaces from associating the result of an operation with a description of an executed action. Even though a REST API is a theoretically a set of synchronous operations to transform basic data resources, in reality users are initiating actions against their account that do not reach their final state immediately. In designing an API, the ability to provide users with relevant information about decisions and actions they take is of utmost importance. Since a separate task representation (e.g. glance) does support providing users with information about the ongoing and final result of specific actions they perform, where a status field does not, we will eventually need to use a task representation in order to properly support our users. Also, the specific detail of whether a resource supports more than one asynchronous operation concurrently (or supports queueing of task operations) is not applicable to this decision. Cloud resources are inherently a distributed system, and individual clients are not able to determine which status is associated with particular actions. For example, consider the following: 1. Client A request operation X be performed 2. Operation X completes successfully 3. Client B requests operation Y be performed 4. Operation Y results in the resource entering an error state 5. Client A checks the status of the resource In this scenario, Client A is unable to report to the user which operation resulted in the resource entering its current error state. If it attempts to report the information according to the information available to it, the user would be under the impression that Operation X caused the resource to enter the error state, with clearly negative impacts on their ability to understand the problem(s) encountered and steps they should take to resolve the situation during their use of the API. Please keep in mind that this message is not related to particular implementation, storage mechanism, or the manner in which clients communicate with the server. I am simply pointing out that the needs of end users can only be properly met by ensuring that particular information is available through the API their applications are using. This is (or should be) the primary driver for design decisions made during the creation of each API. Thank you, Sam Harwell -Original Message- From: Robert Collins [mailto:robe...@robertcollins.net] Sent: Tuesday, June 03, 2014 4:36 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Neutron] Introducing task oriented workflows On 23 May 2014 10:34, Salvatore Orlando wrote: > As most of you probably know already, this is one of the topics discussed > during the Juno summit [1]. > I would like to kick off the discussion in order to move towards a concrete > design. > > Preamble: Considering the meat that's already on the plate for Juno, I'm not > advocating that whatever comes out of this discussion should be put on the > Juno roadmap. However, preparation (or yak shaving) activities that should > be identified as pre-requisite might happen during the Juno time frame > assuming that they won't interfere with other critical or high priority > activities. > This is also a very long post; the TL;DR summary is that I would like to > explore task-oriented communication with the backend and how it should be > reflected in the API - gauging how the community feels about this, and > collecting feedback regarding design, constructs, and related > tools/techniques/technologies. Hi, thanks for writing this up. A few thoughts: - if there can be only one task on a resource at a time, you're essentially forcing all other clients to poll for task completion before coming back to do *their* change. Its kindof a pathological edge case of no in-flight-conflicts :). - Please please please don't embed polling into the design - use webhooks or something similar so that each client (be that Nova, Ironic, Horizon or what-have-you - can get a push response when the thing they want to happen has happened). - I'd think very very carefully about whether you're actually modelling /tasks/ or whether tasks are the implementation and really the core issue is modelling the desired vs obtained resource state - Ironic has a debate going on right now about very much the same problem - the latency involved in some API tasks, and whether the API should return when complete, or when the work his guaranteed to start, or even immediately and maybe the work isn't guaranteed to start. My feeling is that we need to balance ease and correctness of implementation, ease (and ef
Re: [openstack-dev] [Neutron] Introducing task oriented workflows
On 05/22/2014 08:16 PM, Nachi Ueno wrote: Hi Salvatore Thank you for your posting this. IMO, this topic shouldn't be limited for Neutron only. Users wants consistent API between OpenStack project, right? In Nova, a server has task_state, so Neutron should do same way. We're moving away from the simple task_state field in Nova towards a more comprehensive task model. See https://review.openstack.org/#/c/86938/ for the nova-spec around this. 2014-05-22 15:34 GMT-07:00 Salvatore Orlando : As most of you probably know already, this is one of the topics discussed during the Juno summit [1]. I would like to kick off the discussion in order to move towards a concrete design. Preamble: Considering the meat that's already on the plate for Juno, I'm not advocating that whatever comes out of this discussion should be put on the Juno roadmap. However, preparation (or yak shaving) activities that should be identified as pre-requisite might happen during the Juno time frame assuming that they won't interfere with other critical or high priority activities. This is also a very long post; the TL;DR summary is that I would like to explore task-oriented communication with the backend and how it should be reflected in the API - gauging how the community feels about this, and collecting feedback regarding design, constructs, and related tools/techniques/technologies. At the summit a broad range of items were discussed during the session, and most of them have been reported in the etherpad [1]. First, I think it would be good to clarify whether we're advocating a task-based API, a workflow-oriented operation processing, or both. --> About a task-based API In a task-based API, most PUT/POST API operations would return tasks rather than neutron resources, and users of the API will interact directly with tasks. I put an example in [2] to avoid cluttering this post with too much text. As the API operation simply launches a task - the database state won't be updated until the task is completed. Needless to say, this would be a radical change to Neutron's API; it should be carefully evaluated and not considered for the v2 API. Even if it is easily recognisable that this approach has a few benefits, I don't think this will improve usability of the API at all. Indeed this will limit the ability of operating on a resource will a task is in execution on it, and will also require neutron API users to change the paradigm the use to interact with the API; for not mentioning the fact that it would look weird if neutron is the only API endpoint in Openstack operating in this way. For the Neutron API, I think that its operations should still be manipulating the database state, and possibly return immediately after that (*) - a task, or to better say a workflow will then be started, executed asynchronously, and update the resource status on completion. --> On workflow-oriented operations The benefits of it when it comes to easily controlling operations and ensuring consistency in case of failures are obvious. For what is worth, I have been experimenting introducing this kind of capability in the NSX plugin in the past few months. I've been using celery as a task queue, and writing the task management code from scratch - only to realize that the same features I was implementing are already supported by taskflow. I think that all parts of Neutron API can greatly benefit from introducing a flow-based approach. Some examples: - pre/post commit operations in the ML2 plugin can be orchestrated a lot better as a workflow, articulating operations on the various drivers in a graph - operation spanning multiple plugins (eg: add router interface) could be simplified using clearly defined tasks for the L2 and L3 parts - it would be finally possible to properly manage resources' "operational status", as well as knowing whether the actual configuration of the backend matches the database configuration - synchronous plugins might be converted into asynchronous thus improving their API throughput Now, the caveats: - during the sessions it was correctly pointed out that special care is required with multiple producers (ie: api servers) as workflows should be always executed in the correct order - it is probably be advisable to serialize workflows operating on the same resource; this might lead to unexpected situations (potentially to deadlocks) with workflows operating on multiple resources - if the API is asynchronous, and multiple workflows might be queued or in execution at a given time, rolling back the DB operation on failures is probably not advisable (it would not be advisable anyway in any asynchronous framework). If the API instead stays synchronous the revert action for a failed task might also restore the db state for a resource; but I think that keeping the API synchronous missed a bit the point of this whole work - feel free to show your disagreement here! - some neutron workflows are actually initiated by agents; this i
