I'm opinionated because I work with/on Taskflow and had mostly bad experiences with Celery, but here's my $0.02. It's possible that the codebase I inherited just made bad use of Celery or things have improved a lot in the last 18 months, but all I can speak from is my own experience.
Taskflow and Celery at a high level can be used for the same thing, but if anything Taskflow would be considered a replacement for Celery, IMO. We migrated a codebase from Celery to Taskflow because Taskflow gives you a lot more capabilities related to managing the whole process rather than just a bunch of individual async tasks. Some still swear by Celery because they like that they can just throw some decorators on existing methods and execute them asynchronously, and if that's all you need, Taskflow is probably overkill. For us, we wanted a lot more control over the whole flow, including how to handle task failures, being able to run many things in parallel and others in serial, etc. Just conceptually thinking of the process as a series of tasks in a flow makes it a lot easier to reason about what is going on than having to trace throw a bunch of asynchronous functions. We also liked the resiliency guarantees that Taskflow's job board and conductor/worker engine provided more than what Celery had at the time (it didn't seem to guarantee that tasks were executed, and we had fairly often where rabbitmq connections would time out and our process would just not continue, no log, no error, just the next task was never executed). Greg On 1/8/16, 3:19 PM, "Joshua Harlow" <harlo...@fastmail.com> wrote: >So actually they are quite different, (although similar at some level), > >Given that celery isn't really a replacement for taskflow although one >could say, from what I've heard from others, that taskflow is a >super-set of what celery is so taskflow likely can replace parts of >celery (but not vice-versa). > >Feel free to jump on #openstack-state-management IRC channel if u want >to chat in person more about why (it gets into details that might just >be easier to explain in person). > >ESWAR RAO wrote: >> Hi All, >> >> Please let me know whether celery is replacement for taskflow. >> >> As per my understanding, task-flow can break jobs into tasks and execute >> them. >> >> From celery wiki, it also does almost similar behaviour. >> >> I guess in most of openstack components taskflow is widely used. >> Any places where its being replaced with celery ?? >> >> Celery: https://wiki.openstack.org/wiki/Celery >> Distributed: https://wiki.openstack.org/wiki/DistributedTaskManagement >> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow >> >> Thanks >> Eswar >> >> >>_________________________________________________________________________ >>_ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >__________________________________________________________________________ >OpenStack Development Mailing List (not for usage questions) >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev