On Nov 3, 11:53 am, belred <[EMAIL PROTECTED]> wrote:
> On Nov 2, 2:58 pm, Graham Dumpleton <[EMAIL PROTECTED]>
> wrote:
>
>
>
> > On Nov 3, 2:03 am, belred <[EMAIL PROTECTED]> wrote:
>
> > > On Nov 2, 3:29 am, Steve Holden <[EMAIL PROTECTED]> wrote:
>
> > > > belred wrote:
> > > > > i have an wget event in a cron job. the view does some processing
> > > > > which involves calling external websites and databases (takes about 25
> > > > > seconds) and updates some module level dictionary lookup variables
> > > > > (about 7 MB of data) which the rest of the program reads. but
> > > > > unfortunately, only the one apache child process seems to be updated,
> > > > > not all the others. how can i share this module level variable across
> > > > > processes? this lookup happens multiple times per request. but the
> > > > > internal data gets updated nightly.
>
> > > > Aah, so this is the real question! I don't believe Django signals are
> > > > intended to serve needs like this. How does "the rest of the program"
> > > > read the module-level dictionary items? The more explicit you are about
> > > > the exact communication mechanism involved the more likely it is someone
> > > > (probably not me) will be able to help.
>
> > > > regards
> > > > Steve
>
> > > thanks steve,
>
> > > you comments about signals were very clear and should be in the
> > > documentation.
>
> > > when i receive the nightly wget request to update the data, it's
> > > stored in module level dicts which other parts of the program use by
> > > simply importing the data module.
>
> > > import data
>
> > > def view_func(request):
> > > foo = data.lookup['foo']
> > > ...
>
> > > def view_update_data(request):
> > > data.lookup = retrieve_new_data()
> > > ...
>
> > > this works correctly for the child process that receives the update,
> > > but the other child process don't seem to be getting the updates.
>
> > > i thought after i did an update i could os.uptime() thewsgifile
> > > which would cause all of my django project to restart, but it doesn't
> > > seem to be updating my data either.
>
> > If the module level data you are talking about are just transient
> > global variables in the module, then other processes can't see them
> > (or other sub interpreters in same process). For a start read:
>
> > http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
>
> > Written for mod_wsgi, but equally applicable to mod_python for
> > embedded mode description at least. Take note of very last section
> > where it summarises data sharing issues.
>
> > If it is module level data, only solution would be to use mod_wsgi
> > daemon mode and delegate Django instance to run in single process.
>
> > In respect of triggering a restart, if the data is indeed transient,
> > ie., in memory only, how does it get read on startup if you are
> > relying on cron job to trigger its initialisation?
>
> > Graham
>
> in the apps __init__.py it sees if a pickle file exists. if so, it
> read it in and sets the module level variable. if it doesn't exist,
> it call on web services to get new data and process it, setting the
> module level variable and saving a pickle. saving the pickle allows
> the external server to be down as well as my server to restart and
> still use the "last known good data". it also allows my server to
> start instantly instead of 25 second delay while retrieving new data.
> nightly, cron uses wget to call this routine.
If pickle file doesn't exist and so has to contact web service to get
data, for multiprocess web server case where requests may get handled
by different processes at same time just after startup, you just
except that web service may be asked to do same thing multiple times.
Since this is a degenerate case which wouldn't normally happen,
probably okay, but though I would mention it anyway. You would need to
ensure though that the writing of the pickle files by each process
doesn't interfere with each other and allow incomplete file to be read
by another process. Thus, write to unique temporary file and only
rename to actual file when done.
Graham
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
