ESWAR RAO wrote:
Hi Joshua,
Thanks for the response.
As you mentioned, I went through below which has taskflow as celery
front end:
https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine
https://blueprints.launchpad.net/taskflow/+spec/distributed-celery
I think the idea way-back-when was celery would be a backend, not a
front-end, although my memory might be fuzzy in this area (since it was
a few years ago).
I guess if we have a job = task1 + task2 ; if we execute them through a
taskflow parallel pattern, they will be executed in parallel.
Just curious to know that since this threading is built on
eventlets/green threads/python threads they will be affected by GIL and
we may not utilize multi-core capability of our systems.
So that assumes u are using a taskflow engine which is powered by
threads (or greenthreads), there are two types that do not use threads
but actually run out of process:
http://docs.openstack.org/developer/taskflow/engines.html#types
1. http://docs.openstack.org/developer/taskflow/workers.html (runs out
of process, across different machines, using kombu for triggering
start/status... of tasks on other machines)
2. A parallel engine that uses a process pool executor
(https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor)
will then execute tasks using out of process child-processes.
So both of these don't have the same GIL issue (although greenthreads by
there very nature are not affected by the GIL either).
The example (aptly named hello world) shows these different types
(although it doesn't show the worker based one, since that requires more
things to setup) @
https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82
-Josh
It seems in celery, we can have these tasks run as multiple processes so
that they are not affected by GIL.
Right, the above starts to show why taskflow is really a super-set of
celery (for some combination/interpretation of what taskflow or celery do).
Please correct me if my understanding is wrong ??
Thanks
Eswar
On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow <harlo...@fastmail.com
<mailto:harlo...@fastmail.com>> wrote:
I also updated
https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
that said wiki is no longer active (it was an attempt to back a
taskflow engine[1] with celery); although if u are interested in
continuing down this path feel free.
Hopefully that clears up some 'confusion' around that wiki.
[1] http://docs.openstack.org/developer/taskflow/engines.html
Joshua Harlow wrote:
So actually they are quite different, (although similar at some
level),
Given that celery isn't really a replacement for taskflow
although one
could say, from what I've heard from others, that taskflow is a
super-set of what celery is so taskflow likely can replace parts of
celery (but not vice-versa).
Feel free to jump on #openstack-state-management IRC channel if
u want
to chat in person more about why (it gets into details that
might just
be easier to explain in person).
ESWAR RAO wrote:
Hi All,
Please let me know whether celery is replacement for taskflow.
As per my understanding, task-flow can break jobs into tasks
and execute
them.
From celery wiki, it also does almost similar behaviour.
I guess in most of openstack components taskflow is widely used.
Any places where its being replaced with celery ??
Celery: https://wiki.openstack.org/wiki/Celery
Distributed:
https://wiki.openstack.org/wiki/DistributedTaskManagement
TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
Thanks
Eswar
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org
<mailto:openst...@lists.openstack.org>
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org
<mailto:openst...@lists.openstack.org>
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
