Re: What is the best way to implement time-based / cronjob actions in a Django app?

[ringemup]

Thank you, Brian and Shawn, for the further explanations!

On Oct 13, 11:45 pm, Brian Bouterse  wrote:
> RabbitMQ implements a standards based protocol called
> AMQP,
> which provides asynchronous, reliable delivery of messages.  The broker
> simply passes messages around.  Celeryd processes join a message broker
> (RabbitMQ) and pull messages from any number of "queues."  A default queue
> is used if no queue names are specified.  A celeryd worker can be configured
> to pull from any number of queues.
>
> The reliable nature of AMQP based message buses/brokers is that when you
> submit a message into the message broker, you are guaranteed it will not be
> lost, and if a worker is available, it will be delivered.  The asynchronous
> part is that if you submit a message into the broker and there are no
> workers to handle the message, it will queue, reliably with other messages.
>  Once a worker comes online messages will begin being handled.
>
> There is no reason to require the broker to live on a separate machine.  It
> is common in our environment to run a RabbitMQ server process on a server,
> and on that same server configure celeryd worker processes to use localhost
> to connect to the message broker.
>
> To create rabbitmq users I think root permissions are required.  However, I
> do not this root is required for rabbitMQ to be run.  As long as celery has
> the right username/password/server information, and has appropriate
> permissions to run you your task code, you should be able to run celery as
> any non-root user.
>
> In terms of my explanation of celery, it basically describes the data schema
> that is submitted in the AMQP messages which are reliably and asynchronously
> delivered.  For instance, the task function, and all of the submitted
> arguments are contained in the message, and celeryd unpacks these message,
> with this format, and call the task function indicated with those arguments.
>  The return results are serialized and stored in the results database that
> celeryd is configured to work with.  In the case of django-celery these
> settings are found in the django settings.py file, but with celery on its
> own it is in celeryconfig.py  These results can be checked on later by
> task-id which can be easily stored in a plain django model.  We have built
> several analytics and research compute clusters using celery and
> django-celery.
>
> Best,
> Brian
>
>
>
> On Wed, Oct 13, 2010 at 5:07 PM, ringemup  wrote:
> > Thank you for taking the time to explain that, Shawn -- it makes
> > everything a LOT clearer.
>
> > If you could spare the time, I'm curious about a couple of aspects of
> > the architecture:
>
> > - What is the purpose of having a separate broker and daemon?
> > - How does the broker know when to pass the task back to Celery?
> > - Is there a reason other than resource usage for the broker to live
> > on a different machine?
>
> > Also, can this all be run in a shared hosting environment, or are root
> > permissions needed to install Celery and RabbitMQ?
>
> > On Oct 13, 4:43 pm, Shawn Milochik  wrote:
> > > On Oct 13, 2010, at 4:11 PM, ringemup wrote:
>
> > > >> It's surprisingly easy to get set up with nothing more than the
> > tutorial/into for django-celery. If anyone has problems with it I'd be happy
> > to try to assist.
>
> > > > Thanks, I might take you up on that.
>
> > > >> Although getting everything working is fairly easy, in my opinion the
> > docs aren't too clear on how the big picture really works for first-timers.
>
> > > > Yeah, that's a big reason I never tried to use it.  Would you be
> > > > willing to share a high-level overview with us?
>
> > > > Thanks!
>
> > > Okay, so here's how it works, as I understand it. I hope Brian will jump
> > in and correct where necessary.
>
> > > So, as I see it there are three moving parts.
>
> > > 1. Your application.
>
> > >     A. Your application will have, somewhere, some configuration
> > information which allows it to connect to the message broker.
> > >     B. It will also have one or more files containing callable code
> > (probably functions), which are decorated with a Celery decorator. These are
> > referred to as "tasks".
> > >     C. It will have other code which will call these decorated functions
> > when you want things to run asynchronously (in your views, for example).
>
> > > 2. The broker (traditionally RabbitMQ).
>
> > >     A. The broker probably lives on another machine, and runs as a
> > service.
> > >     B. The broker knows nothing about your code or applications.
> > >     C. The broker simply receives messages, holds onto them, and passes
> > them on when requested.
>
> > > 3. The Celery Daemon (the simplest use-case)
>
> > >     A. The Celery daemon is a separate process running on the same
> > machine as your application.
> > >     B. The Celery daemon uses the same config info (probably the same
> > config file) as your ap

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Brian Bouterse]

RabbitMQ implements a standards based protocol called
AMQP,
which provides asynchronous, reliable delivery of messages.  The broker
simply passes messages around.  Celeryd processes join a message broker
(RabbitMQ) and pull messages from any number of "queues."  A default queue
is used if no queue names are specified.  A celeryd worker can be configured
to pull from any number of queues.

The reliable nature of AMQP based message buses/brokers is that when you
submit a message into the message broker, you are guaranteed it will not be
lost, and if a worker is available, it will be delivered.  The asynchronous
part is that if you submit a message into the broker and there are no
workers to handle the message, it will queue, reliably with other messages.
 Once a worker comes online messages will begin being handled.

There is no reason to require the broker to live on a separate machine.  It
is common in our environment to run a RabbitMQ server process on a server,
and on that same server configure celeryd worker processes to use localhost
to connect to the message broker.

To create rabbitmq users I think root permissions are required.  However, I
do not this root is required for rabbitMQ to be run.  As long as celery has
the right username/password/server information, and has appropriate
permissions to run you your task code, you should be able to run celery as
any non-root user.

In terms of my explanation of celery, it basically describes the data schema
that is submitted in the AMQP messages which are reliably and asynchronously
delivered.  For instance, the task function, and all of the submitted
arguments are contained in the message, and celeryd unpacks these message,
with this format, and call the task function indicated with those arguments.
 The return results are serialized and stored in the results database that
celeryd is configured to work with.  In the case of django-celery these
settings are found in the django settings.py file, but with celery on its
own it is in celeryconfig.py  These results can be checked on later by
task-id which can be easily stored in a plain django model.  We have built
several analytics and research compute clusters using celery and
django-celery.

Best,
Brian

On Wed, Oct 13, 2010 at 5:07 PM, ringemup  wrote:

> Thank you for taking the time to explain that, Shawn -- it makes
> everything a LOT clearer.
>
> If you could spare the time, I'm curious about a couple of aspects of
> the architecture:
>
> - What is the purpose of having a separate broker and daemon?
> - How does the broker know when to pass the task back to Celery?
> - Is there a reason other than resource usage for the broker to live
> on a different machine?
>
> Also, can this all be run in a shared hosting environment, or are root
> permissions needed to install Celery and RabbitMQ?
>
>
>
>
>
> On Oct 13, 4:43 pm, Shawn Milochik  wrote:
> > On Oct 13, 2010, at 4:11 PM, ringemup wrote:
> >
> >
> >
> > >> It's surprisingly easy to get set up with nothing more than the
> tutorial/into for django-celery. If anyone has problems with it I'd be happy
> to try to assist.
> >
> > > Thanks, I might take you up on that.
> >
> > >> Although getting everything working is fairly easy, in my opinion the
> docs aren't too clear on how the big picture really works for first-timers.
> >
> > > Yeah, that's a big reason I never tried to use it.  Would you be
> > > willing to share a high-level overview with us?
> >
> > > Thanks!
> >
> > Okay, so here's how it works, as I understand it. I hope Brian will jump
> in and correct where necessary.
> >
> > So, as I see it there are three moving parts.
> >
> > 1. Your application.
> >
> > A. Your application will have, somewhere, some configuration
> information which allows it to connect to the message broker.
> > B. It will also have one or more files containing callable code
> (probably functions), which are decorated with a Celery decorator. These are
> referred to as "tasks".
> > C. It will have other code which will call these decorated functions
> when you want things to run asynchronously (in your views, for example).
> >
> > 2. The broker (traditionally RabbitMQ).
> >
> > A. The broker probably lives on another machine, and runs as a
> service.
> > B. The broker knows nothing about your code or applications.
> > C. The broker simply receives messages, holds onto them, and passes
> them on when requested.
> >
> > 3. The Celery Daemon (the simplest use-case)
> >
> > A. The Celery daemon is a separate process running on the same
> machine as your application.
> > B. The Celery daemon uses the same config info (probably the same
> config file) as your application.
> > C. The Celery daemon polls the broker regularly, looking for tasks.
> > D. When the daemon retrieves a task, it runs it, using the code in
> your application's "tasks" files.
> >
> > Basic working example:
>

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Shawn Milochik]

On Oct 13, 2010, at 5:25 PM, Javier Guerra Giraldez wrote:

> On Wed, Oct 13, 2010 at 3:43 PM, Shawn Milochik  wrote:
>> C. The Celery daemon polls the broker regularly, looking for tasks.
> 
> i hope this isn't polling, but a signal initiated by the broker.
> 
> -- 
> Javier

This could very well be the case. It is implied by the wording here:

http://celeryproject.org/docs/faq.html#why-aren-t-my-remote-control-commands-received-by-all-workers

It sounds like the daemon registers itself with the broker, and the broker 
subsequently pushes messages as appropriate.

Can anyone elaborate?

Thanks,
Shawn

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Shawn Milochik]

On Oct 13, 2010, at 5:07 PM, ringemup wrote:

> Thank you for taking the time to explain that, Shawn -- it makes
> everything a LOT clearer.
> 
> If you could spare the time, I'm curious about a couple of aspects of
> the architecture:
> 
> - What is the purpose of having a separate broker and daemon?
> - How does the broker know when to pass the task back to Celery?
> - Is there a reason other than resource usage for the broker to live
> on a different machine?
> 
> Also, can this all be run in a shared hosting environment, or are root
> permissions needed to install Celery and RabbitMQ?
> 
> 
> 

The whole "messaging" system as implemented in RabbitMQ is very complex (and  
beyond my understanding). So, I'll just say that the broker and daemon are 
separate because it's a lot more flexible this way -- RabbitMQ is used for 
things you and I haven't even thought of. So although a competing software 
product that did the simplified process flow I described could work for 
asynchronous processing in a Django app, but wouldn't be viable for many other 
uses. Because of open source, we have the advantage of an enterprise-level 
piece of software for free, and we can choose to use it even for trivial needs.

The broker has buckets known as 'vhosts.' The default configuration suggested 
by the django-celery docs just has you configure one vhost. The Celery daemon 
is configured to look for a specific vhost. So that's how the broker knows when 
to pass specific tasks to the daemon. You could have many vhosts on one broker. 
In fact, you could have any number of completely unrelated applications using 
the same broker with separate vhosts.

The broker could live on the same machine. I'd say that, in addition to 
resource usage, you'd want the broker on a separate server because any number 
of applications on any number of servers might rely on it. So if a server died, 
or had to be taken down for maintenance, your broker would still be available 
to others. In addition, if other servers were putting tasks in your broker that 
your server was expected to execute, it could then retrieve and execute them 
when it came back up -- no loss of messages.

Shawn

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Javier Guerra Giraldez]

On Wed, Oct 13, 2010 at 3:43 PM, Shawn Milochik  wrote:
> C. The Celery daemon polls the broker regularly, looking for tasks.

i hope this isn't polling, but a signal initiated by the broker.

-- 
Javier

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[ringemup]

Thank you for taking the time to explain that, Shawn -- it makes
everything a LOT clearer.

If you could spare the time, I'm curious about a couple of aspects of
the architecture:

- What is the purpose of having a separate broker and daemon?
- How does the broker know when to pass the task back to Celery?
- Is there a reason other than resource usage for the broker to live
on a different machine?

Also, can this all be run in a shared hosting environment, or are root
permissions needed to install Celery and RabbitMQ?





On Oct 13, 4:43 pm, Shawn Milochik  wrote:
> On Oct 13, 2010, at 4:11 PM, ringemup wrote:
>
>
>
> >> It's surprisingly easy to get set up with nothing more than the 
> >> tutorial/into for django-celery. If anyone has problems with it I'd be 
> >> happy to try to assist.
>
> > Thanks, I might take you up on that.
>
> >> Although getting everything working is fairly easy, in my opinion the docs 
> >> aren't too clear on how the big picture really works for first-timers.
>
> > Yeah, that's a big reason I never tried to use it.  Would you be
> > willing to share a high-level overview with us?
>
> > Thanks!
>
> Okay, so here's how it works, as I understand it. I hope Brian will jump in 
> and correct where necessary.
>
> So, as I see it there are three moving parts.
>
> 1. Your application.
>
>     A. Your application will have, somewhere, some configuration information 
> which allows it to connect to the message broker.
>     B. It will also have one or more files containing callable code (probably 
> functions), which are decorated with a Celery decorator. These are referred 
> to as "tasks".
>     C. It will have other code which will call these decorated functions when 
> you want things to run asynchronously (in your views, for example).
>
> 2. The broker (traditionally RabbitMQ).
>
>     A. The broker probably lives on another machine, and runs as a service.
>     B. The broker knows nothing about your code or applications.
>     C. The broker simply receives messages, holds onto them, and passes them 
> on when requested.
>
> 3. The Celery Daemon (the simplest use-case)
>
>     A. The Celery daemon is a separate process running on the same machine as 
> your application.
>     B. The Celery daemon uses the same config info (probably the same config 
> file) as your application.
>     C. The Celery daemon polls the broker regularly, looking for tasks.
>     D. When the daemon retrieves a task, it runs it, using the code in your 
> application's "tasks" files.
>
> Basic working example:
>
>         1. You have a function in your tasks.py called update_user. It 
> accepts an integer as its only argument, which should be the primary key of a 
> user in your User table. It is decorated by the Celery decorator "task."
>
>         @task
>         def update_user(pk):
>
>             #trivial sample function
>             user = User.objects.get(pk = pk)
>             user.last_login = datetime.now()
>             user.save()
>
>         2. Your application imports your update_user function from your tasks 
> file. One of your views calls it like this:  
> update_user.delay(request.user.pk).  
>         Note that the delay() method is of the Celery task decorator.
>         This call to update_user.delay() returns a UUID which you may store 
> for later retrieval of the results.
>
>         3. Celery passes a serialized version of this function call to the 
> broker. Something like a plain-text "update_user(123)."
>
>         4. The Celery daemon, in its continual polling process, is handed a 
> message containing something like 'update_user(123).' It is aware of the 
> update_user function because it has been configured to use the task files in 
> your application, so it calls your update_user function with the argument 
> 123. At this point your code runs. The celery daemon records the result using 
> whatever method specified in your Celery config file. This could be in 
> MongoDB, passed back to the broker, or several others. Optionally, if the 
> code execution fails, Celery may e-mail you.
>
>        5. (Optional) Your application uses the UUID it received in step 2 at 
> a later time to ascertain the status of the job. If the result was stored 
> with the broker, then it may only be retrieved once; it is considered just a 
> plain-old plain-text "message" to the broker, and after being passed on it is 
> no longer stored. If the result was stored in a database (such as PostgreSQL 
> or MongoDB), then you can request it repeatedly.
>
> I hope this helps, and that others will correct me where I'm blatantly wrong. 
> I have intentionally simplified some things so that the basic flow is more 
> understandable; much more complex setups are possible, especially ones which 
> allow multiple servers to run Celery daemons (and individual servers to run 
> multiple daemons). For example, you may have one server handle communication 
> tasks (such as sending e-mail and SMS messages), while anoth

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Shawn Milochik]

On Oct 13, 2010, at 4:11 PM, ringemup wrote:

> 
>> It's surprisingly easy to get set up with nothing more than the 
>> tutorial/into for django-celery. If anyone has problems with it I'd be happy 
>> to try to assist.
> 
> Thanks, I might take you up on that.
> 
>> Although getting everything working is fairly easy, in my opinion the docs 
>> aren't too clear on how the big picture really works for first-timers.
> 
> Yeah, that's a big reason I never tried to use it.  Would you be
> willing to share a high-level overview with us?
> 
> Thanks!

Okay, so here's how it works, as I understand it. I hope Brian will jump in and 
correct where necessary.

So, as I see it there are three moving parts.

1. Your application.

A. Your application will have, somewhere, some configuration information 
which allows it to connect to the message broker. 
B. It will also have one or more files containing callable code (probably 
functions), which are decorated with a Celery decorator. These are referred to 
as "tasks".
C. It will have other code which will call these decorated functions when 
you want things to run asynchronously (in your views, for example).

2. The broker (traditionally RabbitMQ).

A. The broker probably lives on another machine, and runs as a service.
B. The broker knows nothing about your code or applications.
C. The broker simply receives messages, holds onto them, and passes them on 
when requested.

3. The Celery Daemon (the simplest use-case)

A. The Celery daemon is a separate process running on the same machine as 
your application.
B. The Celery daemon uses the same config info (probably the same config 
file) as your application.
C. The Celery daemon polls the broker regularly, looking for tasks.
D. When the daemon retrieves a task, it runs it, using the code in your 
application's "tasks" files.

Basic working example:

1. You have a function in your tasks.py called update_user. It accepts 
an integer as its only argument, which should be the primary key of a user in 
your User table. It is decorated by the Celery decorator "task."

@task
def update_user(pk):

#trivial sample function
user = User.objects.get(pk = pk)
user.last_login = datetime.now()
user.save()

2. Your application imports your update_user function from your tasks 
file. One of your views calls it like this:  
update_user.delay(request.user.pk).  
Note that the delay() method is of the Celery task decorator.
This call to update_user.delay() returns a UUID which you may store for 
later retrieval of the results.

3. Celery passes a serialized version of this function call to the 
broker. Something like a plain-text "update_user(123)."

4. The Celery daemon, in its continual polling process, is handed a 
message containing something like 'update_user(123).' It is aware of the 
update_user function because it has been configured to use the task files in 
your application, so it calls your update_user function with the argument 123. 
At this point your code runs. The celery daemon records the result using 
whatever method specified in your Celery config file. This could be in MongoDB, 
passed back to the broker, or several others. Optionally, if the code execution 
fails, Celery may e-mail you.

   5. (Optional) Your application uses the UUID it received in step 2 at a 
later time to ascertain the status of the job. If the result was stored with 
the broker, then it may only be retrieved once; it is considered just a 
plain-old plain-text "message" to the broker, and after being passed on it is 
no longer stored. If the result was stored in a database (such as PostgreSQL or 
MongoDB), then you can request it repeatedly.

I hope this helps, and that others will correct me where I'm blatantly wrong. I 
have intentionally simplified some things so that the basic flow is more 
understandable; much more complex setups are possible, especially ones which 
allow multiple servers to run Celery daemons (and individual servers to run 
multiple daemons). For example, you may have one server handle communication 
tasks (such as sending e-mail and SMS messages), while another server handles 
processing of images. It may be beneficial to do one on your application server 
(where your Django app lives), while doing the more resource-intensive stuff 
(such as transcoding video uploads) on another machine.

Shawn



-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Brian Bouterse]

I would be willing to give an overview.  Basically you should:

1) yum or apt-get install rabbitMQ
2) make sure the service is started.
3) Set up your broker by running these 3
commands
.
4)  pip install django-celery
5)  follow steps 1-4 on this
pageto
get your django project to be celery aware
6)  Run at least 1 celery worker like
this
7)  Define and execute a task.  try
these
.

Best,
Brian


On Wed, Oct 13, 2010 at 4:11 PM, ringemup  wrote:

>
> > It's surprisingly easy to get set up with nothing more than the
> tutorial/into for django-celery. If anyone has problems with it I'd be happy
> to try to assist.
>
> Thanks, I might take you up on that.
>
> > Although getting everything working is fairly easy, in my opinion the
> docs aren't too clear on how the big picture really works for first-timers.
>
> Yeah, that's a big reason I never tried to use it.  Would you be
> willing to share a high-level overview with us?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-us...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>


-- 
Brian Bouterse
ITng Services

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[ringemup]

> It's surprisingly easy to get set up with nothing more than the tutorial/into 
> for django-celery. If anyone has problems with it I'd be happy to try to 
> assist.

Thanks, I might take you up on that.

> Although getting everything working is fairly easy, in my opinion the docs 
> aren't too clear on how the big picture really works for first-timers.

Yeah, that's a big reason I never tried to use it.  Would you be
willing to share a high-level overview with us?

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Shawn Milochik]

On Oct 13, 2010, at 1:58 PM, Brian Bouterse wrote:

> I have found cron to be a poor solution for this although it is tempting 
> since it is so easy and will get you started right away.  What I observe 
> happens is, basically someone puts the cron job in crontab, and then a lot of 
> time passes, and folks forget which machine in the cluster is actually in 
> charge of the cron job.  Eventually that machine gets rebuilt or re-deployed, 
> and then folks wonder why stuff mysteriously stops working.  This has 
> happened at least 3 times over the years at different places I have worked 
> at.  (I'm a site reliability engineer).
> 
> Cron is a fine solution, but it's such a single-point solution that it is 
> fragile over time because folks forget about it, and it has a single point of 
> failure.  my 2 cents.
> 
> Brian

I've found exactly this to be the case. I'm in the process of eliminating our 
cron-based activity in favor of a  django-celery solution. We have 
django-celery and RabbitMQ in production and I'm excited about the 
possibilities. Coincidentally, thanks to Brian for his advice and enthusiasm 
for these tools when we met at DjangoCon.

It's surprisingly easy to get set up with nothing more than the tutorial/into 
for django-celery. If anyone has problems with it I'd be happy to try to 
assist. Although getting everything working is fairly easy, in my opinion the 
docs aren't too clear on how the big picture really works for first-timers.

Shawn

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Nick Arnett]

On Wed, Oct 13, 2010 at 4:38 AM, Mattias Linnap  wrote:

> Hi Django users,
>
> I'm building an application with Django, and I need some database
> changes to occur at certain times in the future - independently of any
> web request. Things like cleaning up expired sessions, deleting
> expired user actions, etc.


I use cron to do those kinds of things.   The scripts start with this:

import os
os.environ['DJANGO_SETTINGS_MODULE'] = "settings"
from my_project.models import *

And then they have access to whatever I need.  Actually, instead of
importing *, I usually just import the models I need for that script.

I like keeping all that housekeeping stuff separate from the web app.

Nick

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Brian Bouterse]

I have found cron to be a poor solution for this although it is tempting
since it is so easy and will get you started right away.  What I observe
happens is, basically someone puts the cron job in crontab, and then a lot
of time passes, and folks forget which machine in the cluster is actually in
charge of the cron job.  Eventually that machine gets rebuilt or
re-deployed, and then folks wonder why stuff mysteriously stops working.
 This has happened at least 3 times over the years at different places I
have worked at.  (I'm a site reliability engineer).

Cron is a fine solution, but it's such a single-point solution that it is
fragile over time because folks forget about it, and it has a single point
of failure.  my 2 cents.

Brian

2010/10/13 Ethan Poole 

> Another technique, that I feel should be pointed out, is a custom
> django-admin command:
> http://docs.djangoproject.com/en/dev/howto/custom-management-commands/.
> If you are going to use a crontab to run a Python script, this is by
> far the easiest way to do it.
>
> On 13 Okt, 10:53, ringemup  wrote:
> > You can also use this technique[1] to create a python script that you
> > can call from your crontab.  Basically, you can call anything in
> > Django from any Python script as long as you set the
> > DJANGO_SETTINGS_MODULE environment variable first and have Django,
> > your project, and your apps on your path.
> >
> > [1]http://www.nerdydork.com/django-cron-on-webfaction.html
> >
> > On Oct 13, 10:47 am, Jo  wrote:
> >
> >
> >
> >
> >
> >
> >
> > > On 13 ?.?. 2010, at 18:50, Jonathan Barratt wrote:
> >
> > > > On 13 ?.?. 2010, at 18:38, Mattias Linnap wrote:
> >
> > > >> Hi Django users,
> >
> > > >> I'm building an application with Django, and I need some database
> > > >> changes to occur at certain times in the future - independently of
> any
> > > >> web request. Things like cleaning up expired sessions, deleting
> > > >> expired user actions, etc.
> >
> > > >> What is the best way to implement those?
> >
> > > Having now checked it out for use in my own project, the '"Celery"
> suggestion that's been made to you definitely sounds like the most 'correct'
> way to implement this. Thanks Tom and Brian!
> >
> > > It is, however, as Tom admitted, rather complex. I had another idea
> that I figured was worth throwing out there for your consideration as it's a
> lot easier and faster to implement, though correspondingly not nearly as
> flexible or powerful: just use the *nix command "at." When someone reserves
> an item just execute something along the lines of: (os.popen.) popen('echo
> "COMMAND "'  + PARAMETERS + ' | at now + 2 hours'
> >
> > > This is nowhere near as robust or efficient as I'm sure Celery is, but
> it's so much simpler and quicker that I thought it worth mentioning in case
> you don't have the time to get Celery going right away and need something to
> use for demo\testing purposes in the meantime...
> >
> > > Just a thought,
> > > Jonathan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-us...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>


-- 
Brian Bouterse
ITng Services

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Ethan Poole]

Another technique, that I feel should be pointed out, is a custom
django-admin command: 
http://docs.djangoproject.com/en/dev/howto/custom-management-commands/.
If you are going to use a crontab to run a Python script, this is by
far the easiest way to do it.

On 13 Okt, 10:53, ringemup  wrote:
> You can also use this technique[1] to create a python script that you
> can call from your crontab. ?Basically, you can call anything in
> Django from any Python script as long as you set the
> DJANGO_SETTINGS_MODULE environment variable first and have Django,
> your project, and your apps on your path.
>
> [1]http://www.nerdydork.com/django-cron-on-webfaction.html
>
> On Oct 13, 10:47?am, Jo  wrote:
>
>
>
>
>
>
>
> > On 13 ?.?. 2010, at 18:50, Jonathan Barratt wrote:
>
> > > On 13 ?.?. 2010, at 18:38, Mattias Linnap wrote:
>
> > >> Hi Django users,
>
> > >> I'm building an application with Django, and I need some database
> > >> changes to occur at certain times in the future - independently of any
> > >> web request. Things like cleaning up expired sessions, deleting
> > >> expired user actions, etc.
>
> > >> What is the best way to implement those?
>
> > Having now checked it out for use in my own project, the '"Celery" 
> > suggestion that's been made to you definitely sounds like the most 
> > 'correct' way to implement this. Thanks Tom and Brian!
>
> > It is, however, as Tom admitted, rather complex. I had another idea that I 
> > figured was worth throwing out there for your consideration as it's a lot 
> > easier and faster to implement, though correspondingly not nearly as 
> > flexible or powerful: just use the *nix command "at." When someone reserves 
> > an item just execute something along the lines of: (os.popen.) popen('echo 
> > "COMMAND "' ?+ PARAMETERS + ' | at now + 2 hours'
>
> > This is nowhere near as robust or efficient as I'm sure Celery is, but it's 
> > so much simpler and quicker that I thought it worth mentioning in case you 
> > don't have the time to get Celery going right away and need something to 
> > use for demo\testing purposes in the meantime...
>
> > Just a thought,
> > Jonathan

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[ringemup]

You can also use this technique[1] to create a python script that you
can call from your crontab.  Basically, you can call anything in
Django from any Python script as long as you set the
DJANGO_SETTINGS_MODULE environment variable first and have Django,
your project, and your apps on your path.

[1] http://www.nerdydork.com/django-cron-on-webfaction.html


On Oct 13, 10:47 am, Jo  wrote:
> On 13 ต.ค. 2010, at 18:50, Jonathan Barratt wrote:
>
> > On 13 ต.ค. 2010, at 18:38, Mattias Linnap wrote:
>
> >> Hi Django users,
>
> >> I'm building an application with Django, and I need some database
> >> changes to occur at certain times in the future - independently of any
> >> web request. Things like cleaning up expired sessions, deleting
> >> expired user actions, etc.
>
> >> What is the best way to implement those?
>
> Having now checked it out for use in my own project, the '"Celery" suggestion 
> that's been made to you definitely sounds like the most 'correct' way to 
> implement this. Thanks Tom and Brian!
>
> It is, however, as Tom admitted, rather complex. I had another idea that I 
> figured was worth throwing out there for your consideration as it's a lot 
> easier and faster to implement, though correspondingly not nearly as flexible 
> or powerful: just use the *nix command "at." When someone reserves an item 
> just execute something along the lines of: (os.popen.) popen('echo "COMMAND 
> "'  + PARAMETERS + ' | at now + 2 hours'
>
> This is nowhere near as robust or efficient as I'm sure Celery is, but it's 
> so much simpler and quicker that I thought it worth mentioning in case you 
> don't have the time to get Celery going right away and need something to use 
> for demo\testing purposes in the meantime...
>
> Just a thought,
> Jonathan

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Jo]

On 13 ?.?. 2010, at 18:50, Jonathan Barratt wrote:

> On 13 ?.?. 2010, at 18:38, Mattias Linnap wrote:
> 
>> Hi Django users,
>> 
>> I'm building an application with Django, and I need some database
>> changes to occur at certain times in the future - independently of any
>> web request. Things like cleaning up expired sessions, deleting
>> expired user actions, etc.
>> 
>> What is the best way to implement those?

Having now checked it out for use in my own project, the '"Celery" suggestion 
that's been made to you definitely sounds like the most 'correct' way to 
implement this. Thanks Tom and Brian!

It is, however, as Tom admitted, rather complex. I had another idea that I 
figured was worth throwing out there for your consideration as it's a lot 
easier and faster to implement, though correspondingly not nearly as flexible 
or powerful: just use the *nix command "at." When someone reserves an item just 
execute something along the lines of: (os.popen.) popen('echo "COMMAND "'  + 
PARAMETERS + ' | at now + 2 hours'

This is nowhere near as robust or efficient as I'm sure Celery is, but it's so 
much simpler and quicker that I thought it worth mentioning in case you don't 
have the time to get Celery going right away and need something to use for 
demo\testing purposes in the meantime...

Just a thought,
Jonathan

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Tom Evans]

On Wed, Oct 13, 2010 at 12:38 PM, Mattias Linnap  wrote:
> Hi Django users,
>
> I'm building an application with Django, and I need some database
> changes to occur at certain times in the future - independently of any
> web request. Things like cleaning up expired sessions, deleting
> expired user actions, etc.
>
> What is the best way to implement those?
> 1) Make a special view, like /cron/, that performs all the scheduled
> actions, and use an external script to poll it over HTTP regularly.
> Cons: there might be a request timeout in the web server, so only a
> small number of very fast actions are possible.
> 2) Use standard linux cron, and write a script that calls ./manage.py
> with custom commands. Cons: is it a good idea to use manage.py
> commands in production?
> 3) Try to avoid any time-based database changes, and rewrite the app
> so that data objects know if they are "too old". Cons: makes the rest
> of the app more complex.
> 4) Something else?
>
> If a bit of background might be useful, I'm building an online sales
> app where users can reserve products for up to 2 hours. When the 2
> hours is over, the product is marked as available again. It would be
> possible to compute the availability of each product based on the list
> of reservations instead of caching it at the object, but this would
> require looking through all reservations and sales ever made to know
> the current state of a product.
>
> Thanks for any advice,
>
> Mattias
>

Definitely not #1!

I use #2 for periodic scripts, eg a script runs at 4 AM +? minutes to
clean expired sessions and transient objects. I don't think it would
suit your purposes though, you would have to be periodically running
scripts to see if anything needs going away - not exactly ideal.

#3 is useful in certain scenarios, in this though you are trying to
mimic availability of an item within a stock system. It might work if
you are selling one off items (like a seat in a cinema), but if you
are selling multiples of the same item it could get a little tricky!

I would recommend #4 'something else' :) The something else being
celeryd + rabbitmq + django-celery. RabbitMQ is an enterprise message
queue system that allows you to do awesomeness, like saying 'deliver
this message in 2 hours'. It's a complex system, but it gives you the
power to do complex things.

Cheers

Tom

Celery: http://ask.github.com/celery/getting-started/introduction.html
django-celery: http://celeryproject.org/docs/django-celery/
RabbitMQ: http://www.rabbitmq.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Brian Bouterse]

You really should take a look at
django-celery.
 It is the right tool for this job IMHO.

Brian

On Wed, Oct 13, 2010 at 7:38 AM, Mattias Linnap  wrote:

> Hi Django users,
>
> I'm building an application with Django, and I need some database
> changes to occur at certain times in the future - independently of any
> web request. Things like cleaning up expired sessions, deleting
> expired user actions, etc.
>
> What is the best way to implement those?
> 1) Make a special view, like /cron/, that performs all the scheduled
> actions, and use an external script to poll it over HTTP regularly.
> Cons: there might be a request timeout in the web server, so only a
> small number of very fast actions are possible.
> 2) Use standard linux cron, and write a script that calls ./manage.py
> with custom commands. Cons: is it a good idea to use manage.py
> commands in production?
> 3) Try to avoid any time-based database changes, and rewrite the app
> so that data objects know if they are "too old". Cons: makes the rest
> of the app more complex.
> 4) Something else?
>
> If a bit of background might be useful, I'm building an online sales
> app where users can reserve products for up to 2 hours. When the 2
> hours is over, the product is marked as available again. It would be
> possible to compute the availability of each product based on the list
> of reservations instead of caching it at the object, but this would
> require looking through all reservations and sales ever made to know
> the current state of a product.
>
> Thanks for any advice,
>
> Mattias
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-us...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>


-- 
Brian Bouterse
ITng Services

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What is the best way to implement time-based / cronjob actions in a Django app?

[Jonathan Barratt]

On 13 ต.ค. 2010, at 18:38, Mattias Linnap wrote:

> Hi Django users,
> 
> I'm building an application with Django, and I need some database
> changes to occur at certain times in the future - independently of any
> web request. Things like cleaning up expired sessions, deleting
> expired user actions, etc.
> 
> What is the best way to implement those?

I had need for scheduled events separate from any HTTP requests. I recommend 
you check out django-command-extensions aka django-extensions, 
http://code.google.com/p/django-command-extensions/

It has a jobs system that lets you do cron stuff from within the Django 
paradigm. It's not a perfect match for your needs because it doesn't natively 
provide a way to schedule a job for a given time period from now, but you could 
schedule something every 5 or 15 mins and give users that many minutes grace 
past their official two hours...

Hope that helps,
Jonathan

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

What is the best way to implement time-based / cronjob actions in a Django app?

[Mattias Linnap]

Hi Django users,

I'm building an application with Django, and I need some database
changes to occur at certain times in the future - independently of any
web request. Things like cleaning up expired sessions, deleting
expired user actions, etc.

What is the best way to implement those?
1) Make a special view, like /cron/, that performs all the scheduled
actions, and use an external script to poll it over HTTP regularly.
Cons: there might be a request timeout in the web server, so only a
small number of very fast actions are possible.
2) Use standard linux cron, and write a script that calls ./manage.py
with custom commands. Cons: is it a good idea to use manage.py
commands in production?
3) Try to avoid any time-based database changes, and rewrite the app
so that data objects know if they are "too old". Cons: makes the rest
of the app more complex.
4) Something else?

If a bit of background might be useful, I'm building an online sales
app where users can reserve products for up to 2 hours. When the 2
hours is over, the product is marked as available again. It would be
possible to compute the availability of each product based on the list
of reservations instead of caching it at the object, but this would
require looking through all reservations and sales ever made to know
the current state of a product.

Thanks for any advice,

Mattias

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: cronjob

[Ramiro Morales]

On 9/18/07, patrickk <[EMAIL PROTECTED]> wrote:
>
> now I get:
> ImportError: No module named django.core.management
>
> the directory "django_src" is on my python-path.
>

Read cron documentation; the cronjobs run with a crippled environment
but you can set/add the env vars (PYHTONPATH in this case) you need on
the crontab.

HTH

-- 
 Ramiro Morales

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: cronjob

[Horst Gutmann]

Better also insert it before doing this import. Your cron-manager
probably uses a different shell then your terminal :)

2007/9/18, patrickk <[EMAIL PROTECTED]>:
>
> now I get:
> ImportError: No module named django.core.management
>
> the directory "django_src" is on my python-path.
>
>
> On 18 Sep., 12:43, Christian Joergensen <[EMAIL PROTECTED]> wrote:
> > patrickk wrote:
> > > I just tried to set up a cronjob with plesk and I got this error:
> >
> > > from: can't read /var/mail/django.core.management
> > > /var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
> > > 4: import: command not found
> > > /var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
> > > 5: syntax error near unexpected token `settings'
> > > /var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
> > > 5: `setup_environ(settings)'
> >
> > > Here´s the test-script:
> >
> > > # coding: utf-8
> >
> > You're missing the shebang line:
> >
> > #!/usr/bin/python
> >
> > --
> > Christian Joergensen | Linux, programming or web 
> > consultancyhttp://www.razor.dk | Visit us at:http://www.gmta.info
> >
> >  signature.asc
> > 1KHerunterladen
>
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: cronjob

[Nis Jørgensen]

patrickk skrev:
> now I get:
> ImportError: No module named django.core.management
>
> the directory "django_src" is on my python-path.
>   
It probably isn't on cron's python-path, though. Try adding

PYTHONPATH=/your/python/path

to the top of your crontab.

-- 
Nis Jørgensen
Who is NOT a relative of Christian



--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: cronjob

[patrickk]

now I get:
ImportError: No module named django.core.management

the directory "django_src" is on my python-path.


On 18 Sep., 12:43, Christian Joergensen <[EMAIL PROTECTED]> wrote:
> patrickk wrote:
> > I just tried to set up a cronjob with plesk and I got this error:
>
> > from: can't read /var/mail/django.core.management
> > /var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
> > 4: import: command not found
> > /var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
> > 5: syntax error near unexpected token `settings'
> > /var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
> > 5: `setup_environ(settings)'
>
> > Here´s the test-script:
>
> > # coding: utf-8
>
> You're missing the shebang line:
>
> #!/usr/bin/python
>
> --
> Christian Joergensen | Linux, programming or web 
> consultancyhttp://www.razor.dk | Visit us at:http://www.gmta.info
>
>  signature.asc
> 1KHerunterladen


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: cronjob

[Christian Joergensen]

patrickk wrote:
> I just tried to set up a cronjob with plesk and I got this error:
> 
> from: can't read /var/mail/django.core.management
> /var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
> 4: import: command not found
> /var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
> 5: syntax error near unexpected token `settings'
> /var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
> 5: `setup_environ(settings)'
> 
> Here´s the test-script:
> 
> # coding: utf-8

You're missing the shebang line:

#!/usr/bin/python

-- 
Christian Joergensen | Linux, programming or web consultancy
http://www.razor.dk  | Visit us at: http://www.gmta.info



signature.asc
Description: OpenPGP digital signature

cronjob

[patrickk]

I just tried to set up a cronjob with plesk and I got this error:

from: can't read /var/mail/django.core.management
/var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
4: import: command not found
/var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
5: syntax error near unexpected token `settings'
/var/www/vhosts/mydomain.at/www/scripts/cronjobs/mail_editors.py: line
5: `setup_environ(settings)'

Here´s the test-script:

# coding: utf-8

from django.core.management import setup_environ
import settings
setup_environ(settings)

def mail_movies():
from django.core.mail import send_mail

subject = u"test"
message = u"test"
send_mail(subject, message, "[EMAIL PROTECTED]",
['[EMAIL PROTECTED]'],)


patrick


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---