I guess if you are writing a true "chat" client it may not be the way to go, but if you are looking to run distributed tasks then it should fit the bill. You may need to combine a few technologies together rather than just relying on Celery. I'm not sure you'll have the control over sending a task to a specific "peer" unless you've set up the individual workers using specific routing. It's really good at concurrency but I believe the concurrency is defined when the worker starts up, not on the fly. I could be wrong about that as it isn't a requirement I've had so far.
On Fri, May 30, 2014 at 10:09 AM, Marcus Ottosson <konstrukt...@gmail.com> wrote: > There was actually one more thing, Tony. > > Would you mind have a quick scan through the set of requirements I set out > for Chat, and let me know if you find any that would be tricky or excellent > with Celery? > > > https://github.com/abstractfactory/labs/tree/master/python/messaging/chat#requirements > > > On 30 May 2014 14:43, Marcus Ottosson <konstrukt...@gmail.com> wrote: > >> That is excellent, Tony, thanks! I'll give that a go, look through the >> code, and hopefully that takes care of the routing and task-creation >> questions I had. >> >> To be continued. >> >> >> On 30 May 2014 14:23, Tony Barbieri <great...@gmail.com> wrote: >> >>> I've used Celery pretty extensively at this point. We use it primarily >>> for running various tasks that users don't have permissions to run locally. >>> I basically wrapped it all up into a library that makes it super simple to >>> add new methods with optional tasks associated with them that offers both a >>> json-rpc interface running in a tornado server as well as direct celery >>> access. >>> >>> I only mention it's difficult to reverse-engineer because there is a lot >>> of indirection and dynamic inheritance going on in the internal code. >>> Celery also relies on a few different libraries that are written by the >>> same developer so you have to follow the order of operations through quite >>> a few places. The code is actually really interesting but takes awhile to >>> wrap your head around, and there is a lot of it. However, the public >>> facing API is quite nice and it's very well documented. I've found it to >>> be really nice to work with and it's also pretty robust at this point. >>> There have been quite a few years put into it with a ton of feedback and >>> it seems to be pretty widely used. >>> >>> >>> >>> On Fri, May 30, 2014 at 9:14 AM, Marcus Ottosson <konstrukt...@gmail.com >>> > wrote: >>> >>>> Both. Tasks are to be scheduled via a chat-like interface. Humans send >>>> tasks via a terminal whereas workers build messages programmatically. >>>> >>>> For example, the command.. >>>> >>>> $ order coffee latte --no-milk >>>> >>>> ..should send a task to the swarm who will in turn delegate the task to >>>> workers capable of executing it. >>>> >>>> On the other hand, tasks can be delegated to a group of workers: >>>> >>>> $ peer barista order coffee latte --no-milk >>>> >>>> In which case the swarm will still distribute the task, but in this >>>> case to a pre-determined group of workers. Who also provide an interface to >>>> available services: >>>> >>>> $ peer barista --list-services >>>> >>>> Barista services: >>>> Take order (order) >>>> >>>> I’ve had a brief look at Celery, do you have any experience with it? >>>> How come it’d be difficult to reverse-engineer? Poor docs? Complicated >>>> behaviour? >>>> >>>> Thanks >>>> >>>> >>>> >>>> On 30 May 2014 13:57, Tony Barbieri <great...@gmail.com> wrote: >>>> >>>>> Are you looking to write a chat like program or are you interested in >>>>> having tasks executed by workers on the other side of the "swarm"? Have >>>>> you looked at celery <http://www.celeryproject.org/>? It deals with >>>>> sending tasks to a broker, routing them using exchanges + queues and >>>>> having >>>>> the tasks executed by workers and returning results through a promise. >>>>> It's very neat, but good luck trying to reverse engineer what's going on >>>>> in the code :). >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, May 30, 2014 at 8:12 AM, Marcus Ottosson < >>>>> konstrukt...@gmail.com> wrote: >>>>> >>>>>> Something like this? >>>>>> >>>>>> >>>>>> https://github.com/abstractfactory/labs/blob/master/python/messaging/chat/swarm.py#L57 >>>>>> >>>>>> >>>>>> On 30 May 2014 13:04, Justin Israel <justinisr...@gmail.com> wrote: >>>>>> >>>>>>> You could just register all your different endpoints or types in a >>>>>>> dict where the key is the type and the value is the handler function. >>>>>>> And >>>>>>> each handler function could do the custom logic that transforms the >>>>>>> message >>>>>>> into a conformned message for the output. >>>>>>> On May 30, 2014 11:57 PM, "Marcus Ottosson" <konstrukt...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I’m developing an example program to illustrate the benefits and >>>>>>>> disadvantages of a service-oriented approach to task-distribution; >>>>>>>> mimicking the instant message-approach to routing messages from >>>>>>>> one-to-many >>>>>>>> via a central broker, it’s not quite at the point where pros and cons >>>>>>>> start >>>>>>>> showing their true colors, but getting there. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> First question - Routing multiple inputs to a single output >>>>>>>> >>>>>>>> peer.py represents a running client capable of making requests to >>>>>>>> the server, or swarm.py. Both peer.py and swarm.py handles >>>>>>>> incoming requests/replies via rather long-winded if/else methods: >>>>>>>> >>>>>>>> # Incoming messagesif type == 'letter': >>>>>>>> # do stuff >>>>>>>> elif type == 'service': >>>>>>>> # do stuff >>>>>>>> elif type == 'receipt': >>>>>>>> >>>>>>>> ... >>>>>>>> >>>>>>>> What would be a better/neater approach to routing multiple inputs >>>>>>>> to a single output, in cases where there’d be a large number (50+) of >>>>>>>> branches in logic? >>>>>>>> Go to method >>>>>>>> <https://github.com/abstractfactory/labs/tree/master/python/messaging/chat> >>>>>>>> >>>>>>>> Best, >>>>>>>> Marcus >>>>>>>> >>>>>>>> -- >>>>>>>> *Marcus Ottosson* >>>>>>>> konstrukt...@gmail.com >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Python Programming for Autodesk Maya" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to python_inside_maya+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODUGwFEJDmR0qYEm1YHptPR-%2Ba-4dP_cFDEOWcfF__n3A%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODUGwFEJDmR0qYEm1YHptPR-%2Ba-4dP_cFDEOWcfF__n3A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Python Programming for Autodesk Maya" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to python_inside_maya+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA19uus2-NW7KPOSUf3Fa%2BheZjiOY59JxFqGZD0sCsmfsw%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA19uus2-NW7KPOSUf3Fa%2BheZjiOY59JxFqGZD0sCsmfsw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Marcus Ottosson* >>>>>> konstrukt...@gmail.com >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Python Programming for Autodesk Maya" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to python_inside_maya+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBLKwevT0qa8wFa%2Bz-ujeNnnAZDDfimrxSJKYLxWniC%2BA%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBLKwevT0qa8wFa%2Bz-ujeNnnAZDDfimrxSJKYLxWniC%2BA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -tony >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Python Programming for Autodesk Maya" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to python_inside_maya+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsTwckXni19RSBS9CbFfyGkVJPmZksM13SMfP0yU2%2B6hoQ%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsTwckXni19RSBS9CbFfyGkVJPmZksM13SMfP0yU2%2B6hoQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> *Marcus Ottosson* >>>> konstrukt...@gmail.com >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Python Programming for Autodesk Maya" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to python_inside_maya+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOByLiy6EGEA-oA7Yrp30ZUZcJQqH%3DyMZ7rqgRNNcq0ffg%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOByLiy6EGEA-oA7Yrp30ZUZcJQqH%3DyMZ7rqgRNNcq0ffg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> -tony >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Python Programming for Autodesk Maya" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to python_inside_maya+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsTjVvBcMkBn-eD0TgrYy13eTgbHkbU_nYAP-hHjpTOR7Q%40mail.gmail.com >>> <https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsTjVvBcMkBn-eD0TgrYy13eTgbHkbU_nYAP-hHjpTOR7Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> *Marcus Ottosson* >> konstrukt...@gmail.com >> > > > > -- > *Marcus Ottosson* > konstrukt...@gmail.com > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to python_inside_maya+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBgFMn5Nq0LBPhgHuA3f7JU%3DyHjSvRCW2Nnev6a6WO9xQ%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBgFMn5Nq0LBPhgHuA3f7JU%3DyHjSvRCW2Nnev6a6WO9xQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- -tony -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsROtCpgiqqmw1Tr5LqMJ7kNT7hky%2BW-r--LEcSYmNU%2BuA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
