Re: Suggestions on "Loading" for possible long load-times?

[Andre Terra]

What Matt meant to say is that Celery is *the* framework for handling
background tasks, especially in the Python/Django world.


Cheers,
AT

On Wed, Nov 2, 2011 at 8:55 PM, Matt Schinckel  wrote:

> Russ mentioned it in passing, but I'll point it out for clarity: Celery is
> a great framework for handling background tasks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-users/-/TwtPwcwjzvkJ.
>
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Suggestions on "Loading" for possible long load-times?

[Matt Schinckel]

Russ mentioned it in passing, but I'll point it out for clarity: Celery is 
a great framework for handling background tasks.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/TwtPwcwjzvkJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Suggestions on "Loading" for possible long load-times?

[Kurtis Mullins]

Thanks a lot Russel! It sounds like a pretty reasonable approach to me.

On Tue, Nov 1, 2011 at 7:54 PM, Russell Keith-Magee  wrote:

> On Wed, Nov 2, 2011 at 7:37 AM, Kurtis  wrote:
> > Hey,
> >
> > I've got a part of my site that is just prototyped at the moment. It's
> > not particularly optimized since it's still in the works.
> >
> > I don't think it's going to take a considerable amount of time to load
> > while we have few users. But, when we start migrating our old users to
> > the new system, I can imagine there will be a delay before the
> > calculations are finished and the data is ready to display.
> >
> > Unfortunately, this is real-time calculations so it's not a simple
> > matter of caching it to death. I'm not ready to do any major
> > optimization at this point anyways since it's still a work-in-
> > progress.
> >
> > Anyways, I want to display a "loading" page while the data is
> > prepared. I don't mind if it's a matter of a redirect, done using
> > AJAX, or just using some sort of a Javascript technique that will
> > display the "Loading" dialog while the rest of the data is sent to the
> > browser. What are some suggestions on doing this with Django? I prefer
> > a simple, straight-forward method that won't cause a lot of
> > complication with the existing code base if at all possible.
>
> There's a very common pattern for this kind of activity:
>
>  1) User requests /start page to get some data that is expensive to
> calculate
>
>  2) View for /start creates a Task object that represents the work to
> be executed in the background.
>
>  3) An external process is running, waiting for new Task objects to be
> created. Exactly how this is serviced is up to you, but the important
> part is that the expensive calculation isn't performed in the view
> code -- it's handled by a process external to the web server. Celery
> is a common way to handle this sort of task; you can also knock
> something together quickly with a script running under a cron (run
> once per minute, find all new tasks, process each, then quit. Not
> ideal for production, but it works as a proof of concept).
>
>  4) Once the task has been *created* (Not finished -- just created),
> /start redirects to a /loading page.
>
>  5) /loading contains an ajax lookup that polls the completion status
> of the task
>
>  6) The task executes and completes; on completion, the task is marked
> as "complete".
>
>  7) Once the task is complete, /loading reports the completion, and
> redirects to /finished to display the results of the calculation.
>
> At every step, the web server's interaction is short lived -- create a
> task; check if the task is complete; display the result of the task --
> so it scales well. If you're careful about the design of your task
> processor, you can make sure that this processor scales too; e.g., add
> a second Celery processor for the task, and you've just doubled your
> capacity.
>
> Hope this helps!
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Suggestions on "Loading" for possible long load-times?

[Russell Keith-Magee]

On Wed, Nov 2, 2011 at 7:37 AM, Kurtis  wrote:
> Hey,
>
> I've got a part of my site that is just prototyped at the moment. It's
> not particularly optimized since it's still in the works.
>
> I don't think it's going to take a considerable amount of time to load
> while we have few users. But, when we start migrating our old users to
> the new system, I can imagine there will be a delay before the
> calculations are finished and the data is ready to display.
>
> Unfortunately, this is real-time calculations so it's not a simple
> matter of caching it to death. I'm not ready to do any major
> optimization at this point anyways since it's still a work-in-
> progress.
>
> Anyways, I want to display a "loading" page while the data is
> prepared. I don't mind if it's a matter of a redirect, done using
> AJAX, or just using some sort of a Javascript technique that will
> display the "Loading" dialog while the rest of the data is sent to the
> browser. What are some suggestions on doing this with Django? I prefer
> a simple, straight-forward method that won't cause a lot of
> complication with the existing code base if at all possible.

There's a very common pattern for this kind of activity:

 1) User requests /start page to get some data that is expensive to calculate

 2) View for /start creates a Task object that represents the work to
be executed in the background.

 3) An external process is running, waiting for new Task objects to be
created. Exactly how this is serviced is up to you, but the important
part is that the expensive calculation isn't performed in the view
code -- it's handled by a process external to the web server. Celery
is a common way to handle this sort of task; you can also knock
something together quickly with a script running under a cron (run
once per minute, find all new tasks, process each, then quit. Not
ideal for production, but it works as a proof of concept).

 4) Once the task has been *created* (Not finished -- just created),
/start redirects to a /loading page.

 5) /loading contains an ajax lookup that polls the completion status
of the task

 6) The task executes and completes; on completion, the task is marked
as "complete".

 7) Once the task is complete, /loading reports the completion, and
redirects to /finished to display the results of the calculation.

At every step, the web server's interaction is short lived -- create a
task; check if the task is complete; display the result of the task --
so it scales well. If you're careful about the design of your task
processor, you can make sure that this processor scales too; e.g., add
a second Celery processor for the task, and you've just doubled your
capacity.

Hope this helps!

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Suggestions on "Loading" for possible long load-times?

[Kurtis]

Hey,

I've got a part of my site that is just prototyped at the moment. It's
not particularly optimized since it's still in the works.

I don't think it's going to take a considerable amount of time to load
while we have few users. But, when we start migrating our old users to
the new system, I can imagine there will be a delay before the
calculations are finished and the data is ready to display.

Unfortunately, this is real-time calculations so it's not a simple
matter of caching it to death. I'm not ready to do any major
optimization at this point anyways since it's still a work-in-
progress.

Anyways, I want to display a "loading" page while the data is
prepared. I don't mind if it's a matter of a redirect, done using
AJAX, or just using some sort of a Javascript technique that will
display the "Loading" dialog while the rest of the data is sent to the
browser. What are some suggestions on doing this with Django? I prefer
a simple, straight-forward method that won't cause a lot of
complication with the existing code base if at all possible.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.