Re: Cron vs event triggered action

[David De La Harpe Golden]

On Sat, Dec 19, 2009 at 06:28:23AM -0800, Tim Daniel wrote:
> David, Celery sounds really good, thanks for the tip, I'll have a
> deeper look into it as soon as I've got some time, if I can't get it

> one question: Is it capable of running on every web server
> that supports Python?

Well, it's kind of independent of the web server bits: In principle, one could
use celery plus django orm for some project that doesn't involve any web serving
at all (celery still depends on django-as-an-orm-layer-and-stuff, but not
django-as-a-web-application-server). See:
http://ask.github.com/celery/faq.html#can-i-use-celery-without-django

So it's really "where can celery run". Unix/Linux/MacOSX and probably Windows
servers where you have shell and the ability to kick off a celeryd daemon, sure.
I'd be less clear on strange setups where django itself or parts of
django can be coaxed into working for some value of working ("cloud" stuff,
jython...).

> or are there any other requirements or things to
> think about?

Well, the message queue and carrot (the binding to the message queue celery 
sits on top). 
The message queue is a service you will have to run. It's easier than it may 
sound 
if on on linux, since rabbitmq is probably in your distro repository. and celery
can be easy_installed, pulling in carrot.

("rabbit" probably explaining the rabbit-food naming theme...)

(Once you have the message queue and carrot available, you may also find them
useful in themselves for tieing stuff together
http://ask.github.com/carrot/introduction.html#examples )

Oh yeah, other thing:
If you're using python <2.6, you'll need to use the python multiprocessing
backport, and you currently have to watch out for an annoying bug - if you 
naively 
install what is still the current release, you'll 
get one with the bug, either use an earlier or patched version.
http://github.com/ask/celery/issues/closed/#issue/24
http://code.google.com/p/python-multiprocessing/issues/detail?id=18
(I wish they'd do another release...)
http://code.google.com/p/python-multiprocessing/issues/detail?id=21

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[creecode]

Hello Tim,

On Dec 19, 6:28 am, Tim Daniel  wrote:

> Brian & Creecode Django Custom management commands are really the same
> as this:
>
> from django.core.management import setup_environ
> import settings
> setup_environ(settings)
>
> or am I wrong? I'm just running the scripts with these three lines on
> top (even before the other import statements).

They both do the above.  Its just that the custom management commands
give you a little extra with it's built-in switches and such.  Use
whatever system works for your needs.

Toodle-l...
creecode

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[Tim Daniel]

David, Celery sounds really good, thanks for the tip, I'll have a
deeper look into it as soon as I've got some time, if I can't get it
up for this project I'll study it's details and put it up later maybe
for a new release or a new project. For what I've seen until now it
seems to be perfect, because you completly control the whole process,
once a job has completed it even returns you a backtrace in case of
failure or a success message. Only the setup/installation looks quite
complex, one question: Is it capable of running on every web server
that supports Python? or are there any other requirements or things to
think about?

John your solution is very smart too, but sometimes it could cause
some overhead, if I do actions while the user is offline or seeing
another section, the user won't notice it, but if you do it inside a
request the user may has to wait a little longer, it depends on the
complexity of the operation you want to do.

Brian & Creecode Django Custom management commands are really the same
as this:

from django.core.management import setup_environ
import settings
setup_environ(settings)

or am I wrong? I'm just running the scripts with these three lines on
top (even before the other import statements).

The apscheduler looks like a good and simple implementation for A,
thanks Guilherme.

And finally Mateus the at(if it's available) command would be a very
easy solution for A too.

On 15 dic, 11:42, David De La Harpe Golden
 wrote:
> Tim Daniel wrote:
> > So how can I implement solution B? Is there a posibility to create a
> >cronon a user action that executes only one time?
>
> > NOTE: I don't want to rely on a thread that should stay alive for two
> > hours ore more inside the server memory.
>
> Well, celery uses a "celeryd" daemon process and a message queue, so
> that is a thread staying alive om the server, but it's a completely
> separate and manageable process from your web server:
>
> http://ask.github.com/celery/introduction.html
>
> It may look a little complex, but it really makes all sorts of long
> running and scheduled tasks easy and well-integrated with django.
>
> FWIW, doing something two hours after someeventis basically a 
> one-liner:http://ask.github.com/celery/userguide/executing.html#eta-and-countdown

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[David De La Harpe Golden]

Tim Daniel wrote:

> So how can I implement solution B? Is there a posibility to create a
> cron on a user action that executes only one time?
> 
> NOTE: I don't want to rely on a thread that should stay alive for two
> hours ore more inside the server memory.


Well, celery uses a "celeryd" daemon process and a message queue, so
that is a thread staying alive om the server, but it's a completely
separate and manageable process from your web server:

http://ask.github.com/celery/introduction.html

It may look a little complex, but it really makes all sorts of long
running and scheduled tasks easy and well-integrated with django.

FWIW, doing something two hours after some event is basically a one-liner:
http://ask.github.com/celery/userguide/executing.html#eta-and-countdown


--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[john2095]

Just to throw it out there... forget about trying to make something
happen every ten minutes; or whatever your margin for error is.
Instead focus on just testing to see whether two hours have passed
when you come to use it. Either lazy (C), or diligent (D)...

Solution C:
Do nothing at all until someone comes asking for it: Have two hours
passed since the create_time? Yes. Trash. Sorry, you were too slow.

Solution D:
Every time someone comes asking for that object: Wait, let me just
trash everything i have which is over two hours old... ok. Now may I
help you? Oh, I'm sorry I don't have that anymore, maybe it expired.

(C) could lead to lots of dead stuff lying around whereas (D) could
lead to a lot of unnecessary overhead.
But for lightly loaded apps they'll do the job without the hassle.

Just for fun.

On Dec 13, 9:58 am, Tim Daniel  wrote:
> Just want to figure out if there is a smarter solution for handling
> the following problem:
>
> 1. An action is performed by a user (Normal django behaviour handling
> a request and giving a response).
> 2. Two hours later I want an automatic action to be done.
>
> Solution A: Have a datetime field with an expiry date and say every 10
> minutes a cron job checks the DB table for expired entries and
> performs the programed action.
>
> Solution B: Have an event triggered cronjob that only executes once
> and is created from Django(Python), after the 2 hours passed it
> performs the programmed action only on the required entry.
>
> So how can I implement solution B? Is there a posibility to create a
> cron on a user action that executes only one time?
>
> NOTE: I don't want to rely on a thread that should stay alive for two
> hours ore more inside the server memory.

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[Guilherme Cavalcanti]

I deployed on on EC2 using apache + mod_wsgi and it worked.

My APS setup is as follow:

def start_notificator():
sched = Scheduler()
sched.add_interval_job(notificator.send_notification, seconds=10)
sched.daemonic = True
t = Thread(target=fire_scheduler, args=[sched], name="Notificator")
t.setDaemon(True)
t.start()

notificator.send_notification() is responsible for searching expired
"events" entries on database and sending them via e-mail/twitter/sms.

fire_scheduler just starts the scheduler (sched, defined up there) and
sets up some configs.


On Dec 14, 7:23 pm, Javier Guerra  wrote:
> On Mon, Dec 14, 2009 at 5:12 PM, Guilherme Cavalcanti
>
>  wrote:
> > If are you going to choose A, take a look on Advanced Python Schedule
> > (http://apscheduler.nextday.fi/). It's a python module that let you
> > schedule some script to be executed periodically, it really makes the
> > job easier.
>
> sounds really great; but, from the very first paragraph:
>
>     APScheduler is a light but powerful in-process task scheduler
>
> the "in-process" part can make it very powerful while easy to use; but
> can it run _after_ Django has returned the response to the web server?
>  IOW: does APScheduler run 'outside' the request-process-response
> loop?  or maybe it works with flup but not mod_wsgi, or the other way
> around?
>
> in any case, it's worth some checking
>
> --
> Javier

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[mateusz.szulc]

Have you consider the linux at command instead of cron?

Example:
echo "touch /home/mateusz/my.name.is.at" | at 02:30

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[creecode]

Hello all,

On Dec 14, 3:57 pm, Brian Neal  wrote:

> http://docs.djangoproject.com/en/dev/howto/custom-management-commands/

Thanks for answering Brian!  Saved me having to do that part
myself! :-)

Tim,  I did it the python script way at first but have been moving
over the the custom management command way.  The cmcs have more
overhead but I like that there is a defined way to plug commands into
my apps.  And there are some interesting built-in commands line
switches which can be seen by doing a ./manage.py help.

Toodle-lo.
creecode

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[Brian Neal]

On Dec 14, 12:27 pm, Tim Daniel  wrote:
> ...
>
> Creecode what do you mean with "custom management commands"? I just
> wrote a simple python script that can be called from bash like this
> python myscript.py, inside it I set up the enviroment to be able to
> call my models and perform actions using the simplicity of the django
> webframework.

http://docs.djangoproject.com/en/dev/howto/custom-management-commands/

The advantage here is that the environment is already setup for you.
You do have to look at some of the django source to figure out what
needs to be done, but it is pretty self-explanatory.

BTW, I use cron to fire off custom management commands.

Regards,
BN

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[Javier Guerra]

On Mon, Dec 14, 2009 at 5:12 PM, Guilherme Cavalcanti
 wrote:
> If are you going to choose A, take a look on Advanced Python Schedule
> (http://apscheduler.nextday.fi/). It's a python module that let you
> schedule some script to be executed periodically, it really makes the
> job easier.


sounds really great; but, from the very first paragraph:

APScheduler is a light but powerful in-process task scheduler

the "in-process" part can make it very powerful while easy to use; but
can it run _after_ Django has returned the response to the web server?
 IOW: does APScheduler run 'outside' the request-process-response
loop?  or maybe it works with flup but not mod_wsgi, or the other way
around?

in any case, it's worth some checking


-- 
Javier

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[Guilherme Cavalcanti]

If are you going to choose A, take a look on Advanced Python Schedule
(http://apscheduler.nextday.fi/). It's a python module that let you
schedule some script to be executed periodically, it really makes the
job easier.

On Dec 14, 3:27 pm, Tim Daniel  wrote:
> Thanks for answering, I think I'm going to go with 'A' too.
>
> Guilherme I've had already looked after django-cron but don't know why
> but it doesn't seem to work well, I've tried to increment a simple
> counter in the database every 30 seconds and it doesn't work(using the
> development server with django 1.1.1.). Also I don't see features
> there to execute a job only once or to kill it.
>
> On django-jits all their sites seem to be offline : "Page
> "InstallAndConfig" Not Found", and looking at the comments on the
> first and only issue/ticket it seems to be an abandoned and not
> finished project. So it isn't a reliable solution.
>
> And django-notification is not the thing I'm looking for because it
> seems to be just for sending notifications to a user in the moment.
>
> So I'm still looking for a solution with better performance and less
> overhead than 'A'.
>
> Creecode what do you mean with "custom management commands"? I just
> wrote a simple python script that can be called from bash like this
> python myscript.py, inside it I set up the enviroment to be able to
> call my models and perform actions using the simplicity of the django
> webframework. There I just check the entries on my cron-db table, and
> if its time I perform the corresponding action (maybe having a type
> field defining actions like sendmail or delete_account).
>
> On 14 dic, 12:30, Guilherme Cavalcanti 
> wrote:
>
>
>
> > Tim, recently I've used the solution A too.
>
> > Take a look on these plugins:
>
> >http://code.google.com/p/django-cron/http://code.google.com/p/django-...
>
> > Pay attention specially on django-jits, it uses a little different
> > approach based on your argument (threads).
>
> > Excuse my bad english.
>
> > On Dec 12, 7:58 pm, Tim Daniel  wrote:
>
> > > Just want to figure out if there is a smarter solution for handling
> > > the following problem:
>
> > > 1. An action is performed by a user (Normal django behaviour handling
> > > a request and giving a response).
> > > 2. Two hours later I want an automatic action to be done.
>
> > > Solution A: Have a datetime field with an expiry date and say every 10
> > > minutes a cron job checks the DB table for expired entries and
> > > performs the programed action.
>
> > > Solution B: Have an event triggered cronjob that only executes once
> > > and is created from Django(Python), after the 2 hours passed it
> > > performs the programmed action only on the required entry.
>
> > > So how can I implement solution B? Is there a posibility to create a
> > > cron on a user action that executes only one time?
>
> > > NOTE: I don't want to rely on a thread that should stay alive for two
> > > hours ore more inside the server memory.

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[Tim Daniel]

Thanks for answering, I think I'm going to go with 'A' too.

Guilherme I've had already looked after django-cron but don't know why
but it doesn't seem to work well, I've tried to increment a simple
counter in the database every 30 seconds and it doesn't work(using the
development server with django 1.1.1.). Also I don't see features
there to execute a job only once or to kill it.

On django-jits all their sites seem to be offline : "Page
"InstallAndConfig" Not Found", and looking at the comments on the
first and only issue/ticket it seems to be an abandoned and not
finished project. So it isn't a reliable solution.

And django-notification is not the thing I'm looking for because it
seems to be just for sending notifications to a user in the moment.

So I'm still looking for a solution with better performance and less
overhead than 'A'.

Creecode what do you mean with "custom management commands"? I just
wrote a simple python script that can be called from bash like this
python myscript.py, inside it I set up the enviroment to be able to
call my models and perform actions using the simplicity of the django
webframework. There I just check the entries on my cron-db table, and
if its time I perform the corresponding action (maybe having a type
field defining actions like sendmail or delete_account).

On 14 dic, 12:30, Guilherme Cavalcanti 
wrote:
> Tim, recently I've used the solution A too.
>
> Take a look on these plugins:
>
> http://code.google.com/p/django-cron/http://code.google.com/p/django-jits/http://github.com/jtauber/django-notification/blob/master/docs/usage.txt
>
> Pay attention specially on django-jits, it uses a little different
> approach based on your argument (threads).
>
> Excuse my bad english.
>
> On Dec 12, 7:58 pm, Tim Daniel  wrote:
>
> > Just want to figure out if there is a smarter solution for handling
> > the following problem:
>
> > 1. An action is performed by a user (Normal django behaviour handling
> > a request and giving a response).
> > 2. Two hours later I want an automatic action to be done.
>
> > Solution A: Have a datetime field with an expiry date and say every 10
> > minutes a cron job checks the DB table for expired entries and
> > performs the programed action.
>
> > Solution B: Have an event triggered cronjob that only executes once
> > and is created from Django(Python), after the 2 hours passed it
> > performs the programmed action only on the required entry.
>
> > So how can I implement solution B? Is there a posibility to create a
> > cron on a user action that executes only one time?
>
> > NOTE: I don't want to rely on a thread that should stay alive for two
> > hours ore more inside the server memory.

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[Guilherme Cavalcanti]

Tim, recently I've used the solution A too.

Take a look on these plugins:

http://code.google.com/p/django-cron/
http://code.google.com/p/django-jits/
http://github.com/jtauber/django-notification/blob/master/docs/usage.txt

Pay attention specially on django-jits, it uses a little different
approach based on your argument (threads).

Excuse my bad english.

On Dec 12, 7:58 pm, Tim Daniel  wrote:
> Just want to figure out if there is a smarter solution for handling
> the following problem:
>
> 1. An action is performed by a user (Normal django behaviour handling
> a request and giving a response).
> 2. Two hours later I want an automatic action to be done.
>
> Solution A: Have a datetime field with an expiry date and say every 10
> minutes a cron job checks the DB table for expired entries and
> performs the programed action.
>
> Solution B: Have an event triggered cronjob that only executes once
> and is created from Django(Python), after the 2 hours passed it
> performs the programmed action only on the required entry.
>
> So how can I implement solution B? Is there a posibility to create a
> cron on a user action that executes only one time?
>
> NOTE: I don't want to rely on a thread that should stay alive for two
> hours ore more inside the server memory.

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Cron vs event triggered action

[creecode]

Hello Tim,

On Dec 12, 2:58 pm, Tim Daniel  wrote:

> Solution A: Have a datetime field with an expiry date and say every 10
> minutes a cron job checks the DB table for expired entries and
> performs the programed action.

I've used solution A in combination with custom management commands.
Works pretty well.  I send emails to the user for notification.

> Solution B: Have an event triggered cronjob that only executes once
> and is created from Django(Python), after the 2 hours passed it
> performs the programmed action only on the required entry.
>
> So how can I implement solution B? Is there a posibility to create a
> cron on a user action that executes only one time?

You might see if you could use the at command in combination with a
custom management command.

Let us know if you go with B and how it turns out.

Toodle-loo..
creecode

--

You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.