On 07/09/2014 11:39 AM, Clint Byrum wrote:
Excerpts from Yuriy Taraday's message of 2014-07-09 03:36:00 -0700:
On Tue, Jul 8, 2014 at 11:31 PM, Joshua Harlow <harlo...@yahoo-inc.com>
wrote:
I think clints response was likely better than what I can write here, but
I'll add-on a few things,
How do you write such code using taskflow?
@asyncio.coroutine
def foo(self):
result = yield from some_async_op(...)
return do_stuff(result)
The idea (at a very high level) is that users don't write this;
What users do write is a workflow, maybe the following (pseudocode):
# Define the pieces of your workflow.
TaskA():
def execute():
# Do whatever some_async_op did here.
def revert():
# If execute had any side-effects undo them here.
TaskFoo():
...
# Compose them together
flow = linear_flow.Flow("my-stuff").add(TaskA("my-task-a"),
TaskFoo("my-foo"))
I wouldn't consider this composition very user-friendly.
I find it extremely user friendly when I consider that it gives you
clear lines of delineation between "the way it should work" and "what
to do when it breaks."
Agreed.
snip...
Sorry but the code above is nothing like the code that Josh shared. When
create_network(project) fails, how do we revert its side effects? If we
want to resume this flow after reboot, how does that work?
Exactly.
I understand that there is a desire to write everything in beautiful
python yields, try's, finally's, and excepts. But the reality is that
python's stack is lost the moment the process segfaults, power goes out
on that PDU, or the admin rolls out a new kernel.
Yup.
If we embed taskflow deep in the code, we get those things, and we can
treat tasks as coroutines and let taskflow's event loop be asyncio just
the same. If we embed asyncio deep into the code, we don't get any of
the high level functions and we get just as much code churn.
++
There's no limit to coroutine usage. The only problem is the library that
would bind everything together.
In my example run_task will have to be really smart, keeping track of all
started tasks, results of all finished ones, skipping all tasks that have
already been done (and substituting already generated results).
But all of this is doable. And I find this way of declaring workflows way
more understandable than whatever would it look like with Flow.add's
The way the flow is declared is important, as it leads to more isolated
code. The single place where the flow is declared in Josh's example means
that the flow can be imported, the state deserialized and inspected,
and resumed by any piece of code: an API call, a daemon start up, an
admin command, etc.
Right, this is the main point. We are focusing so much on eventlet vs.
asyncio, and in doing so we are missing the big picture in how we think
about the flows of related tasks in our code. Taskflow makes that big
picture thinking possible, and is what I believe our focus should be on.
If someone hates seeing eventlet's magic masking of async I/O and wants
to see Py3K-clean yields, then I think that work belongs in the Taskflow
engines modules, and not inside Nova directly.
Besides Py3K support and predictive yield points, I haven't seen any
other valid arguments for spending a bunch of time moving from eventlet
to asyncio, and certainly no arguments that address the real
architectural problems inside Nova: that we continue to think at too low
a level and instead of writing code that naturally groups related sets
of tasks into workflows, we think instead about how to properly yield
from one coroutine to another. The point of eventlet, I thought, was to
hide the low-level stuff so that developers could focus on higher-level
(and more productive) abstractions. Introducing asyncio contructs into
the higher level code like Nova and Neutron seems to be a step in the
wrong direction, IMHO. I'd rather see a renewed focus on getting
Taskflow incorporated into Nova.
Best,
-jay
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
