Re: [web2py] how to start Celery worker in web2py
hello. Can you unsubscribe me please? I am not currenly a web2py user. But maybe in future. Please remove my email from you list for now. Kind regards, Alexandra On Sun, Nov 4, 2018 at 8:10 AM James O' Driscoll wrote: > I used DAL(uri, folder, import_models=True) in the task.py function and > this does the job. > > If possible can someone explain if: > > 1.This lightweight connect to celery is preferred over > https://code.google.com/archive/p/web2py-celery/. > 2.Is running the worker using task manager as a windows service the > best practice? > > Regards, > James > > > On Saturday, November 3, 2018 at 4:46:43 PM UTC+10, James O' Driscoll > wrote: >> >> I have implemented the above, the gen_url function is working but the >> add_user function is not (receiving the error db is not defined from >> tasks.py.) >> >> Regards, >> James >> > -- > Resources: > - http://web2py.com > - http://web2py.com/book (Documentation) > - http://github.com/web2py/web2py (Source code) > - https://code.google.com/p/web2py/issues/list (Report Issues) > --- > You received this message because you are subscribed to the Google Groups > "web2py-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to web2py+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [web2py] how to start Celery worker in web2py
I used DAL(uri, folder, import_models=True) in the task.py function and this does the job. If possible can someone explain if: 1.This lightweight connect to celery is preferred over https://code.google.com/archive/p/web2py-celery/. 2.Is running the worker using task manager as a windows service the best practice? Regards, James On Saturday, November 3, 2018 at 4:46:43 PM UTC+10, James O' Driscoll wrote: > > I have implemented the above, the gen_url function is working but the > add_user function is not (receiving the error db is not defined from > tasks.py.) > > Regards, > James > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [web2py] how to start Celery worker in web2py
I have implemented the above, the gen_url function is working but the add_user function is not (receiving the error db is not defined.) Regards, James -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [web2py] how to start Celery worker in web2py
> > >>> celery worker -A tasks > File "", line 1 > celery worker -A tasks > ^ > SyntaxError: invalid syntax > The above is not Python code -- it belongs at the command line. Also, I notice you often add replies to threads that are several years old. Unless you are directly replying to that thread, you should instead just start a new post with an appropriate title -- that way we don't develop very long threads that branch off into only tangentially related questions (if it is helpful for context, your new post can include links to old threads). Anthony -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [web2py] how to start Celery worker in web2py
trying to learn web2py with celery, but an error occured
*- install celery*
source activate test2
pip install celery
*- start redis server from source install (stable version 4.0.9)*
./src/redis-server
*- start web2py (stable version 2.16.1 on python 2.7)*
source activate test2
python ~/python/web2py/web2py.py --nogui --no-banner -a a -i 0.0.0.0 -p 8000
*- create new web2py app named : celery*
*- create modules*
modules/w2p_celery.py
from celery import Celery
mycelery = Celery('tasks', broker='redis://localhost:6379/0',
backend='redis://localhost:6379/0')
*- create models*
*models/thecelerymodel.py*
from w2p_celery import mycelery
celery = mycelery
@celery.task(name='tasks.gen_url')
def gen_url(x):
return A(x, _href=URL('rule_the_world'))
@celery.task(name='tasks.add_user')
def add_user():
try:
db.auth_user.insert(first_name='John')
db.commit()
except:
db.rollback()
*- create tasks.py in the same folder as web2py.py*
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import gluon.widget#forgot: why this is apparently the only way to
fix custom imports ?
from gluon.shell import env
from gluon import current
from celery import Celery
def make_celery(app):
celery = Celery('tasks', broker='redis://localhost:6379/0',
backend='redis://localhost:6379/0')
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
_env = env(a=app, import_models=True)
globals().update(_env)
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
celery = make_celery('celery') #be sure that you write this correctly
@celery.task(name='tasks.gen_url')
def gen_url(x):
return A(x, _href=URL(x, 'rule_the_world'))
@celery.task(name='tasks.add_user')
def add_user(): #yes, for the love of your apps, wrap all db operations!
try:
db.auth_user.insert(first_name='miao')
db.commit()
except:
db.rollback()
*- start celery*
cd ~/python/web2py/
source activate test2
python
>>> import celery
>>> celery worker -A tasks
File "", line 1
celery worker -A tasks
^
SyntaxError: invalid syntax
any idea?
thanks and best regards,
stifan
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [web2py] how to start Celery worker in web2py
if it's right, and I'm not sure it is since I couldn't get the to the inner circle of knowledge required to run such a thing and tell all the others people, THIS is the way ^_^ On Wednesday, March 13, 2013 5:09:18 AM UTC+1, Massimo Di Pierro wrote: This belongs to a blog post! -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] how to start Celery worker in web2py
I'm interested in a robust, widely-adopted scheduler. The current web2py Scheduler is clearly changing very rapidly, which is great, for now I want a scheduler that is mature. Can anyone answer my original question -- how have you gotten Celery workers to run with web2py? On Thursday, March 7, 2013 9:47:25 AM UTC-8, Massimo Di Pierro wrote: What I want to know is what do you think celery buys you that the built-in scheduler does not provide? Celery is faster at transferring messages from the application t the workers and vice versa but normally when you want to run background tasks you have different bottle-necks: computation cycles of background tasks and database access from the tasks. In this respect I do not see celery being any better than the built-in scheduler. Actually the built-in scheduler makes your life easier by exposing the web2py environment to workers which is something celery would not be able to do. On Wednesday, 6 March 2013 21:11:59 UTC-6, rochacbruno wrote: I also would like to see Celery, Solr, Elastic Search and other fantastic tools working with web2py! I think this is an important issue and I am sure it is completely easy and possible to make it. I personally do not like to use the built-in scheduler, so I am using python-rq (Redis Queue) for some production sites and it works very well and offer almost all celery functionalities. Maybe someone can follow my python-rq[1] example and create a wen2py-celery tutorial [1]http://rochacbruno.com.br/web2py-and-redis-queue/ I dont think web2py needs to always reinvent the wheel so I would like to see more integrations. wish list: Whoosh (WIP) Solr (maybe a haystack clone for web2py) ElasticSearch Celery Thumbor Neo4J On Wed, Mar 6, 2013 at 10:47 PM, Eric S ericea...@gmail.com wrote: I would like to use Celery in my web2py application, but I'm having trouble with how to start a Celery worker (I know there is a web2py scheduler but I would like to use Celery). To start a custom scheduler in web2py I would use: python web2py.py -S appName -M -R worker.py Celery workers, however, are launched from the command line such as with the following command, which won't easily substitute into 'worker.py' above: celery -A tasks worker --loglevel=info Has anyone had success integrating web2py and Celery? How do you (robustly) get around this problem? Thanks, Eric -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] how to start Celery worker in web2py
more than celery we'd need a generalized web2py create the context recipe running tasks defined in modules is easy. running tasks outside web2py that needs the usual environment is a PITA On Tuesday, March 12, 2013 6:17:47 PM UTC+1, Eric S wrote: I'm interested in a robust, widely-adopted scheduler. The current web2py Scheduler is clearly changing very rapidly, which is great, for now I want a scheduler that is mature. Can anyone answer my original question -- how have you gotten Celery workers to run with web2py? On Thursday, March 7, 2013 9:47:25 AM UTC-8, Massimo Di Pierro wrote: What I want to know is what do you think celery buys you that the built-in scheduler does not provide? Celery is faster at transferring messages from the application t the workers and vice versa but normally when you want to run background tasks you have different bottle-necks: computation cycles of background tasks and database access from the tasks. In this respect I do not see celery being any better than the built-in scheduler. Actually the built-in scheduler makes your life easier by exposing the web2py environment to workers which is something celery would not be able to do. On Wednesday, 6 March 2013 21:11:59 UTC-6, rochacbruno wrote: I also would like to see Celery, Solr, Elastic Search and other fantastic tools working with web2py! I think this is an important issue and I am sure it is completely easy and possible to make it. I personally do not like to use the built-in scheduler, so I am using python-rq (Redis Queue) for some production sites and it works very well and offer almost all celery functionalities. Maybe someone can follow my python-rq[1] example and create a wen2py-celery tutorial [1]http://rochacbruno.com.br/web2py-and-redis-queue/ I dont think web2py needs to always reinvent the wheel so I would like to see more integrations. wish list: Whoosh (WIP) Solr (maybe a haystack clone for web2py) ElasticSearch Celery Thumbor Neo4J On Wed, Mar 6, 2013 at 10:47 PM, Eric S ericea...@gmail.com wrote: I would like to use Celery in my web2py application, but I'm having trouble with how to start a Celery worker (I know there is a web2py scheduler but I would like to use Celery). To start a custom scheduler in web2py I would use: python web2py.py -S appName -M -R worker.py Celery workers, however, are launched from the command line such as with the following command, which won't easily substitute into 'worker.py' above: celery -A tasks worker --loglevel=info Has anyone had success integrating web2py and Celery? How do you (robustly) get around this problem? Thanks, Eric -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] how to start Celery worker in web2py
Is it not possible launch a Celery worker that has access to the web2py environment? This is possible for custom schedulers with commands such as the following -- why would it not be available to Celery workers? python web2py.py -S appName -M -R worker.py On Tuesday, March 12, 2013 10:17:47 AM UTC-7, Eric S wrote: I'm interested in a robust, widely-adopted scheduler. The current web2py Scheduler is clearly changing very rapidly, which is great, for now I want a scheduler that is mature. Can anyone answer my original question -- how have you gotten Celery workers to run with web2py? On Thursday, March 7, 2013 9:47:25 AM UTC-8, Massimo Di Pierro wrote: What I want to know is what do you think celery buys you that the built-in scheduler does not provide? Celery is faster at transferring messages from the application t the workers and vice versa but normally when you want to run background tasks you have different bottle-necks: computation cycles of background tasks and database access from the tasks. In this respect I do not see celery being any better than the built-in scheduler. Actually the built-in scheduler makes your life easier by exposing the web2py environment to workers which is something celery would not be able to do. On Wednesday, 6 March 2013 21:11:59 UTC-6, rochacbruno wrote: I also would like to see Celery, Solr, Elastic Search and other fantastic tools working with web2py! I think this is an important issue and I am sure it is completely easy and possible to make it. I personally do not like to use the built-in scheduler, so I am using python-rq (Redis Queue) for some production sites and it works very well and offer almost all celery functionalities. Maybe someone can follow my python-rq[1] example and create a wen2py-celery tutorial [1]http://rochacbruno.com.br/web2py-and-redis-queue/ I dont think web2py needs to always reinvent the wheel so I would like to see more integrations. wish list: Whoosh (WIP) Solr (maybe a haystack clone for web2py) ElasticSearch Celery Thumbor Neo4J On Wed, Mar 6, 2013 at 10:47 PM, Eric S ericea...@gmail.com wrote: I would like to use Celery in my web2py application, but I'm having trouble with how to start a Celery worker (I know there is a web2py scheduler but I would like to use Celery). To start a custom scheduler in web2py I would use: python web2py.py -S appName -M -R worker.py Celery workers, however, are launched from the command line such as with the following command, which won't easily substitute into 'worker.py' above: celery -A tasks worker --loglevel=info Has anyone had success integrating web2py and Celery? How do you (robustly) get around this problem? Thanks, Eric -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] how to start Celery worker in web2py
cause I know exactly how to create a web2py environment in my own module but for the life of me I can't figure out how to apply the same method to a celery worker (if you're not a fan of magic in web2py, celery workers instances do a LT of magic behind the scenes) On Tuesday, March 12, 2013 10:19:45 PM UTC+1, Eric S wrote: Is it not possible launch a Celery worker that has access to the web2py environment? This is possible for custom schedulers with commands such as the following -- why would it not be available to Celery workers? python web2py.py -S appName -M -R worker.py On Tuesday, March 12, 2013 10:17:47 AM UTC-7, Eric S wrote: I'm interested in a robust, widely-adopted scheduler. The current web2py Scheduler is clearly changing very rapidly, which is great, for now I want a scheduler that is mature. Can anyone answer my original question -- how have you gotten Celery workers to run with web2py? On Thursday, March 7, 2013 9:47:25 AM UTC-8, Massimo Di Pierro wrote: What I want to know is what do you think celery buys you that the built-in scheduler does not provide? Celery is faster at transferring messages from the application t the workers and vice versa but normally when you want to run background tasks you have different bottle-necks: computation cycles of background tasks and database access from the tasks. In this respect I do not see celery being any better than the built-in scheduler. Actually the built-in scheduler makes your life easier by exposing the web2py environment to workers which is something celery would not be able to do. On Wednesday, 6 March 2013 21:11:59 UTC-6, rochacbruno wrote: I also would like to see Celery, Solr, Elastic Search and other fantastic tools working with web2py! I think this is an important issue and I am sure it is completely easy and possible to make it. I personally do not like to use the built-in scheduler, so I am using python-rq (Redis Queue) for some production sites and it works very well and offer almost all celery functionalities. Maybe someone can follow my python-rq[1] example and create a wen2py-celery tutorial [1]http://rochacbruno.com.br/web2py-and-redis-queue/ I dont think web2py needs to always reinvent the wheel so I would like to see more integrations. wish list: Whoosh (WIP) Solr (maybe a haystack clone for web2py) ElasticSearch Celery Thumbor Neo4J On Wed, Mar 6, 2013 at 10:47 PM, Eric S ericea...@gmail.com wrote: I would like to use Celery in my web2py application, but I'm having trouble with how to start a Celery worker (I know there is a web2py scheduler but I would like to use Celery). To start a custom scheduler in web2py I would use: python web2py.py -S appName -M -R worker.py Celery workers, however, are launched from the command line such as with the following command, which won't easily substitute into 'worker.py' above: celery -A tasks worker --loglevel=info Has anyone had success integrating web2py and Celery? How do you (robustly) get around this problem? Thanks, Eric -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] how to start Celery worker in web2py
ok, I think I got it . most of it is cumbersome and fruit of multiple
reiteration (e.g. lots of trials and errors). There's no way in hell to put
up celery docs pertaining the particular usecase in web2py.
Pleease, watch it carefully, may burn your house to the ground.
@Bruno: maybe the same thing can be applied to rq (I still have problems
with running that one with functions not defined in models outside web2py
environment)
Chosen broker and result backend -- redis (absolutely no time to install
rabbitmq, sorry)
1st issue: models are executed at every request, so a celery instance
defined in models continues to instantiate new connections to Redis.
Although redis is fast as hell, it just mean wasting resources (a single
redis instance makes use by default of a connection pool, but if the object
is recycled and istantiated at every request the pool doesn't survice). So,
celery instance must be defined in a *module* and imported
1st step, hence -- create modules/w2p_celery.py with
from celery import Celery
mycelery = Celery('tasks', broker='redis://localhost:6379/0', backend=
'redis://localhost:6379/0')
2nd issue unresolved: the default task decorator computes a name based on
the current execution instance (for newcomers to web2py, __restricted__)
Celery has a task decorator that takes a name parameter, so it's quite easy
to patch (even so, I'd really like to be able to customize that.
Suggestions ?)
However, if you put this in models/thecelerymodel.py
from w2p_celery import mycelery
celery = mycelery
@celery.task(name='tasks.gen_url')
def gen_url(x):
return A(x, _href=URL('rule_the_world'))
@celery.task(name='tasks.add_user')
def add_user():
try:
db.auth_user.insert(first_name='John')
db.commit()
except:
db.rollback()
you'll have two tasks registered as 'tasks.add' and 'tasks.add_user'
3rd step: f***k the world. having something runnable by a celery
worker. This part was the absolute nightmare of the last two evenings. I
really didn't get the problems in it, but seems that task code is
somewhat-where-when shipped around (or, my nightmares were in fact the
reality)
Given that, let's assume that copying the tasks defined before elsewhere
won't get you killed in action (i.e. yes, I'd like too to have celery
workers figuring out where to pick something from, but I just couldn't get
there)
3rd step -- create a tasks.py in the same folder as web2py.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import gluon.widget#forgot: why this is apparently the only way to
fix custom imports ?
from gluon.shell import env
from gluon import current
from celery import Celery
def make_celery(app):
celery = Celery('tasks', broker='redis://localhost:6379/0', backend=
'redis://localhost:6379/0')
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
_env = env(a=app, import_models=True)
globals().update(_env)
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
celery = make_celery('yourappname') #be sure that you write this correctly
@celery.task(name='tasks.gen_url')
def gen_url(x):
return A(x, _href=URL(x, 'rule_the_world'))
@celery.task(name='tasks.add_user')
def add_user(): #yes, for the love of your apps, wrap all db operations!
try:
db.auth_user.insert(first_name='miao')
db.commit()
except:
db.rollback()
now, this **seems** to work without issues, e.g. in a controller you can do
without harm
def myfunction():
async = add_user.delay()
return dict(async=async)
and start an instance of celery, cding into web2py.py folder and doing
celery worker -A tasks
Again, this is not the recommended way nor the documented somewhere one.
It is just the first **iteration** that didn't lend in exceptions all over
the place.
If someone wants to pick this up, go to ask (solem) and get it
corrected/revised, I'll be glad to have spent 6 hours for the web2py
community :P
@PS on the robust-widely-adopted argument: celery vs web2py's scheduler is
like comparing web2py to web.py. They have VERY DIFFERENT goals in mind and
celery is by far the most complete task queue solution out there (even
comparing other programming languages task queues). With that in mind (i.e.
even Niphlod in the need of a gigantic solution for out-of-the-band
processes chooses celery), celery broke a few times backward compatibility.
Web2py's scheduler got new features every iteration and just broke code
when it had no API whatsoever to retrieve task results (has just changed
the db schema once, to accomodate all engines reserved keywords)
--
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
Re: [web2py] how to start Celery worker in web2py
This belongs to a blog post!
On Tuesday, 12 March 2013 17:23:21 UTC-5, Niphlod wrote:
ok, I think I got it . most of it is cumbersome and fruit of multiple
reiteration (e.g. lots of trials and errors). There's no way in hell to put
up celery docs pertaining the particular usecase in web2py.
Pleease, watch it carefully, may burn your house to the ground.
@Bruno: maybe the same thing can be applied to rq (I still have problems
with running that one with functions not defined in models outside web2py
environment)
Chosen broker and result backend -- redis (absolutely no time to install
rabbitmq, sorry)
1st issue: models are executed at every request, so a celery instance
defined in models continues to instantiate new connections to Redis.
Although redis is fast as hell, it just mean wasting resources (a single
redis instance makes use by default of a connection pool, but if the object
is recycled and istantiated at every request the pool doesn't survice). So,
celery instance must be defined in a *module* and imported
1st step, hence -- create modules/w2p_celery.py with
from celery import Celery
mycelery = Celery('tasks', broker='redis://localhost:6379/0', backend=
'redis://localhost:6379/0')
2nd issue unresolved: the default task decorator computes a name based on
the current execution instance (for newcomers to web2py, __restricted__)
Celery has a task decorator that takes a name parameter, so it's quite
easy to patch (even so, I'd really like to be able to customize that.
Suggestions ?)
However, if you put this in models/thecelerymodel.py
from w2p_celery import mycelery
celery = mycelery
@celery.task(name='tasks.gen_url')
def gen_url(x):
return A(x, _href=URL('rule_the_world'))
@celery.task(name='tasks.add_user')
def add_user():
try:
db.auth_user.insert(first_name='John')
db.commit()
except:
db.rollback()
you'll have two tasks registered as 'tasks.add' and 'tasks.add_user'
3rd step: f***k the world. having something runnable by a celery
worker. This part was the absolute nightmare of the last two evenings. I
really didn't get the problems in it, but seems that task code is
somewhat-where-when shipped around (or, my nightmares were in fact the
reality)
Given that, let's assume that copying the tasks defined before elsewhere
won't get you killed in action (i.e. yes, I'd like too to have celery
workers figuring out where to pick something from, but I just couldn't get
there)
3rd step -- create a tasks.py in the same folder as web2py.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import gluon.widget#forgot: why this is apparently the only way
to fix custom imports ?
from gluon.shell import env
from gluon import current
from celery import Celery
def make_celery(app):
celery = Celery('tasks', broker='redis://localhost:6379/0', backend=
'redis://localhost:6379/0')
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
_env = env(a=app, import_models=True)
globals().update(_env)
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
celery = make_celery('yourappname') #be sure that you write this correctly
@celery.task(name='tasks.gen_url')
def gen_url(x):
return A(x, _href=URL(x, 'rule_the_world'))
@celery.task(name='tasks.add_user')
def add_user(): #yes, for the love of your apps, wrap all db operations!
try:
db.auth_user.insert(first_name='miao')
db.commit()
except:
db.rollback()
now, this **seems** to work without issues, e.g. in a controller you can
do without harm
def myfunction():
async = add_user.delay()
return dict(async=async)
and start an instance of celery, cding into web2py.py folder and doing
celery worker -A tasks
Again, this is not the recommended way nor the documented somewhere one.
It is just the first **iteration** that didn't lend in exceptions all over
the place.
If someone wants to pick this up, go to ask (solem) and get it
corrected/revised, I'll be glad to have spent 6 hours for the web2py
community :P
@PS on the robust-widely-adopted argument: celery vs web2py's scheduler is
like comparing web2py to web.py. They have VERY DIFFERENT goals in mind and
celery is by far the most complete task queue solution out there (even
comparing other programming languages task queues). With that in mind (i.e.
even Niphlod in the need of a gigantic solution for out-of-the-band
processes chooses celery), celery broke a few times backward compatibility.
Web2py's scheduler got new features every iteration and just broke code
when it had no API whatsoever to retrieve task results (has just changed
the db schema once, to accomodate all engines reserved keywords)
--
---
You received this message
Re: [web2py] how to start Celery worker in web2py
+1 . Scheduler is a great tool because its feature packed and exploits what is at hand in a normal deployment environment (and it's the best shot at replacing cron likes). The minute I had a redis-backed scheduler at hand (its there, sitting on my disk) I was kinda sad, because what makes the scheduler great is the ease of interaction with the database All users needed to switch to a complete different toolset to queue tasks, at that point... better to rely on something battle-tested (always a problem in web2py environment, too many newbies and little or few testers). For a realtime offloading tool Celery it's probably the best solution out there (kinda harder to config, but hey), pyres it's my favourite and rq and huey are littler in comparison but much easier to config and solve the 80% of the problems. I used them only as a mere task dispatchers/processors, i.e. outside the web world. PS: Bruno, I tried to follow your code when you originally posted it, but it had 2 problems : - 1: every request ends up building a connection towards Redis (I think a singleton is needed) - 2: I remember some problems about scheduling functions defined in models (i.e. adapting to the crazy execution model of web2py) As soon as I'll get home I'll try again, but is there a chance that in the meantime those problems have been already fixed ? -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] how to start Celery worker in web2py
What I want to know is what do you think celery buys you that the built-in scheduler does not provide? Celery is faster at transferring messages from the application t the workers and vice versa but normally when you want to run background tasks you have different bottle-necks: computation cycles of background tasks and database access from the tasks. In this respect I do not see celery being any better than the built-in scheduler. Actually the built-in scheduler makes your life easier by exposing the web2py environment to workers which is something celery would not be able to do. On Wednesday, 6 March 2013 21:11:59 UTC-6, rochacbruno wrote: I also would like to see Celery, Solr, Elastic Search and other fantastic tools working with web2py! I think this is an important issue and I am sure it is completely easy and possible to make it. I personally do not like to use the built-in scheduler, so I am using python-rq (Redis Queue) for some production sites and it works very well and offer almost all celery functionalities. Maybe someone can follow my python-rq[1] example and create a wen2py-celery tutorial [1]http://rochacbruno.com.br/web2py-and-redis-queue/ I dont think web2py needs to always reinvent the wheel so I would like to see more integrations. wish list: Whoosh (WIP) Solr (maybe a haystack clone for web2py) ElasticSearch Celery Thumbor Neo4J On Wed, Mar 6, 2013 at 10:47 PM, Eric S ericea...@gmail.com javascript: wrote: I would like to use Celery in my web2py application, but I'm having trouble with how to start a Celery worker (I know there is a web2py scheduler but I would like to use Celery). To start a custom scheduler in web2py I would use: python web2py.py -S appName -M -R worker.py Celery workers, however, are launched from the command line such as with the following command, which won't easily substitute into 'worker.py' above: celery -A tasks worker --loglevel=info Has anyone had success integrating web2py and Celery? How do you (robustly) get around this problem? Thanks, Eric -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com javascript:. For more options, visit https://groups.google.com/groups/opt_out. -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] how to start Celery worker in web2py
I would like to use Celery in my web2py application, but I'm having trouble with how to start a Celery worker (I know there is a web2py scheduler but I would like to use Celery). To start a custom scheduler in web2py I would use: python web2py.py -S appName -M -R worker.py Celery workers, however, are launched from the command line such as with the following command, which won't easily substitute into 'worker.py' above: celery -A tasks worker --loglevel=info Has anyone had success integrating web2py and Celery? How do you (robustly) get around this problem? Thanks, Eric -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] how to start Celery worker in web2py
I also would like to see Celery, Solr, Elastic Search and other fantastic tools working with web2py! I think this is an important issue and I am sure it is completely easy and possible to make it. I personally do not like to use the built-in scheduler, so I am using python-rq (Redis Queue) for some production sites and it works very well and offer almost all celery functionalities. Maybe someone can follow my python-rq[1] example and create a wen2py-celery tutorial [1]http://rochacbruno.com.br/web2py-and-redis-queue/ I dont think web2py needs to always reinvent the wheel so I would like to see more integrations. wish list: Whoosh (WIP) Solr (maybe a haystack clone for web2py) ElasticSearch Celery Thumbor Neo4J On Wed, Mar 6, 2013 at 10:47 PM, Eric S ericearlsm...@gmail.com wrote: I would like to use Celery in my web2py application, but I'm having trouble with how to start a Celery worker (I know there is a web2py scheduler but I would like to use Celery). To start a custom scheduler in web2py I would use: python web2py.py -S appName -M -R worker.py Celery workers, however, are launched from the command line such as with the following command, which won't easily substitute into 'worker.py' above: celery -A tasks worker --loglevel=info Has anyone had success integrating web2py and Celery? How do you (robustly) get around this problem? Thanks, Eric -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
