This belongs to a blog post! On Tuesday, 12 March 2013 17:23:21 UTC-5, Niphlod wrote: > > ok, I think I got it ..... most of it is cumbersome and fruit of multiple > reiteration (e.g. lots of trials and errors). There's no way in hell to put > up celery docs pertaining the particular usecase in web2py. > Pleeeeeease, watch it carefully, may burn your house to the ground. > @Bruno: maybe the same thing can be applied to rq (I still have problems > with running that one with functions not defined in models outside web2py > environment) > Chosen broker and result backend --> redis (absolutely no time to install > rabbitmq, sorry) > > 1st issue: models are executed at every request, so a celery instance > defined in models continues to instantiate new connections to Redis. > Although redis is fast as hell, it just mean wasting resources (a single > redis instance makes use by default of a connection pool, but if the object > is recycled and istantiated at every request the pool doesn't survice). So, > celery instance must be defined in a *module* and imported > 1st step, hence --> create modules/w2p_celery.py with > > from celery import Celery > mycelery = Celery('tasks', broker='redis://localhost:6379/0', backend= > 'redis://localhost:6379/0') > > 2nd issue unresolved: the default task decorator computes a name based on > the current execution instance (for newcomers to web2py, __restricted__) > Celery has a task decorator that takes a name parameter, so it's quite > easy to patch (even so, I'd really like to be able to customize that. > Suggestions ?) > However, if you put this in models/thecelerymodel.py > from w2p_celery import mycelery > celery = mycelery > @celery.task(name='tasks.gen_url') > def gen_url(x): > return A(x, _href=URL('rule_the_world')) > > @celery.task(name='tasks.add_user') > def add_user(): > try: > db.auth_user.insert(first_name='John') > db.commit() > except: > db.rollback() > > you'll have two tasks registered as 'tasks.add' and 'tasks.add_user' > > 3rd step: f*******k the world. having something runnable by a celery > worker. This part was the absolute nightmare of the last two evenings. I > really didn't get the problems in it, but seems that task "code" is > somewhat-where-when shipped around (or, my nightmares were in fact the > reality....) > Given that, let's assume that copying the tasks defined before elsewhere > won't get you killed in action (i.e. yes, I'd like too to have celery > workers figuring out where to pick something from, but I just couldn't get > there) > 3rd step --> create a tasks.py in the same folder as web2py.py > > #!/usr/bin/env python > # -*- coding: utf-8 -*- > import gluon.widget #forgot: why this is apparently the only way > to fix custom imports ? > from gluon.shell import env > from gluon import current > from celery import Celery > > def make_celery(app): > celery = Celery('tasks', broker='redis://localhost:6379/0', backend= > 'redis://localhost:6379/0') > TaskBase = celery.Task > class ContextTask(TaskBase): > abstract = True > def __call__(self, *args, **kwargs): > _env = env(a=app, import_models=True) > globals().update(_env) > return TaskBase.__call__(self, *args, **kwargs) > celery.Task = ContextTask > return celery > > celery = make_celery('yourappname') #be sure that you write this correctly > > @celery.task(name='tasks.gen_url') > def gen_url(x): > return A(x, _href=URL(x, 'rule_the_world')) > > @celery.task(name='tasks.add_user') > def add_user(): #yes, for the love of your apps, wrap all db operations! > try: > db.auth_user.insert(first_name='miao') > db.commit() > except: > db.rollback() > > now, this **seems** to work without issues, e.g. in a controller you can > do without harm > > def myfunction(): > async = add_user.delay() > return dict(async=async) > > and start an instance of celery, cding into web2py.py folder and doing > >>> celery worker -A tasks > > Again, this is not the recommended way nor the "documented somewhere" one. > It is just the first **iteration** that didn't lend in exceptions all over > the place. > If someone wants to pick this up, go to ask (solem) and get it > corrected/revised, I'll be glad to have spent 6 hours for the web2py > community :P > > @PS on the robust-widely-adopted argument: celery vs web2py's scheduler is > like comparing web2py to web.py. They have VERY DIFFERENT goals in mind and > celery is by far the most complete task queue solution out there (even > comparing other programming languages task queues). With that in mind (i.e. > even Niphlod in the need of a gigantic solution for out-of-the-band > processes chooses celery), celery broke a few times backward compatibility. > Web2py's scheduler got new features every iteration and just broke code > when it had no API whatsoever to retrieve task results (has just changed > the db schema once, to accomodate all engines reserved keywords) >
-- --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.