Hey Tony, I haven't used celery for more than a super simple addition as a local background task queue to a Django server, so I thought I would ask specifically related to the render farm concept that Marcus mentioned, and celery's granularity. Does celery give you the control to define the scheduler? In the case where you want to direct tasks to the best fitting workers, does celery take into account available resources on the workers vs requested resources of the task? In situations where you want to direct to a subset of workers or a specific worker, does that equal a new queue? Do workers have to be preconfigured with "slots" and do tasks consume N available slots?
I can see how well suited celery would be for generalized task distribution, but I am curious how much support is included vs how much has to be developed as another layer to handle the type of profile of a render queue, where the focus is trying to reach and maintain maximum resource utilization and task throughput, and the least amount of task re-queues. And also with the need for priority sorting. But again maybe that is something that has to be a scheduler that sits in front of it actually going into the celery queues. Just curious what your thoughts are on the celery architecture for this kind of application, from your experience with it? On Sat, May 31, 2014 at 6:43 AM, Tony Barbieri <great...@gmail.com> wrote: > Thanks, chad! > > We haven't used it for that although it's a very interesting idea. We've > just started writing some tools for crawling hierarchies, using Celery to > distribute the work sounds like a very interesting idea... > > Also good to know about the persistent option, I'll give that a look if > we end up needing it. For now keeping long-term historical data for the > types of tasks we're running hasn't been necessary. Typically the > historical data is only used for fixing immediate issues that may crop up. > > > On Fri, May 30, 2014 at 2:31 PM, Chad Dombrova <chad...@gmail.com> wrote: > >> cool stuff, tony. do you use celery to do multi-worker distributed disk >> crawls, like finding all the files below a given directory? this is >> another use case that I'm very interested in. >> >> also, btw, there is an option in flower to persist the tasks states to >> disk, but there's a simple bug that needs to be fixed to get it to work. >> >> chad. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Python Programming for Autodesk Maya" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to python_inside_maya+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7H35CPhkT%3Dz_n6B4zA0AHS8L9Cw69mf8JDtgf-fTFBZUw%40mail.gmail.com >> <https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7H35CPhkT%3Dz_n6B4zA0AHS8L9Cw69mf8JDtgf-fTFBZUw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > -tony > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to python_inside_maya+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsRthNKfAXDXN3L194c4844shEgOYBisMEaqcBkxhDi%2BwA%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsRthNKfAXDXN3L194c4844shEgOYBisMEaqcBkxhDi%2BwA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA25255sU6%2BMQZZyye-OsgwRXTXd-%3Db3imd6mFOH3jraEQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
