"Daryl Spitzer" <[EMAIL PROTECTED]> writes: >> If you need to, you could build a persistent daemon that occasionally >> checked for new data to import (say, once every 30 seconds) and would >> spawn a separate thread for each bit of data to be imported. > > I think this is what I'll do. I can't quickly check whether there is > new data to import, so I think I'll have the daemon pickle the data > into a temporary file (and when done, replace the existing file). My > view can then unpickle from that file. (I'll probably have it run > every 10 minutes or so, unless it gets a signal to start ASAP.) > > The daemon has to read from the Django project's database; is it > possible (and safe) to have it use Django's ORM to do so?
We haven't had any problems yet. I think you should be fine. >> ...or, the easy way out, just provide a page that provides a status, >> where your user can refresh as they like. > > I'll have the daemon include a datetime value in the pickled data > which can be displayed in the page, and add a refresh button that will > trigger the signal I describe above. > > Thanks Christopher. > > -- > Daryl > > > On Tue, Apr 15, 2008 at 10:17 AM, Christopher Allan Webber > <[EMAIL PROTECTED]> wrote: >> >> We developed something that did something similar here. The import >> process spawned a daemon which imported the data. However, in our >> system only one import could be done at a time (that was actually part >> of the spec, not bad programming!), so spawning one daemon per import >> wasn't such a bad idea. >> >> If you need to, you could build a persistent daemon that occasionally >> checked for new data to import (say, once every 30 seconds) and would >> spawn a separate thread for each bit of data to be imported. >> >> Of course, you'll have to develop a good workflow so that either your >> user is notified on that page when the import is completed (possibly >> through some ajax-y interface either by polling or using a Comet >> system) or, the easy way out, just provide a page that provides a >> status, where your user can refresh as they like. All depends on your >> situation. >> >> Hope that helps, >> Christopher Allan Webber >> >> >> >> "Daryl Spitzer" <[EMAIL PROTECTED]> writes: >> >> > I have a view that takes approximately 5 minutes (plus or minus 4 or >> > so) to put together data (culled from other sources, such as our SCM >> > and a web service that wraps our bug database) to be passed on to a >> > template. Some form of caching may (mostly) solve this problem, but >> > I'm worried about two things: >> > >> > 1) The unlucky user who is the first to go to this page after the >> > cached data has expired will have a long wait. >> > >> > 2) What happens when other users go to other pages, or this page, >> > while the view is gathering data? >> > >> > I don't have any experience deploying Django projects (yet) nor do I >> > have any with Apache. But from what I've read so far, I'm under the >> > impression that Apache can be configured to run multiple processes, >> > each with its own invocation of Django...right? (These processes >> > will, of course, be unaware of each other unless I take steps to >> > synchronize them.) >> > >> > I think the proper solution is to have a separate process periodically >> > put together the data that my view does now, and serialize (pickle) it >> > or put it in the database. Then my view will just read and display the >> > most-recently generated data. >> > >> > Do you agree? Do you have any advice? Should I run this separate >> > process from within Django (somehow), or as a cronjob? >> > >> > -- >> > Daryl Spitzer >> > >> > >> >> > >> > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
