On Apr 8, 5:43 pm, Andy Baker <andy...@gmail.com> wrote:
> Wow! So Python will eat RAM until restarted? That changes the way I am
> thinking about this problem.
>
> So the trick will be tweaking the number of processes and the number of
> requests allowed between a restart. I know one item requests a big chunk of
> memory is dynamic PDF generation.
>
> So lets say this is the real RAM pig and eats up 80mb. If I've got 3
> processes that could account for 240mb on it's own which you say will never
> get given back? However - it should never exceed this as once grabbed that
> RAM will be available to my app.
>
> Even caching won't reduce this as there is always the possibility on all
> three processes being hit at the same time with nothing in the cache.
>
> So my memory usage will always statistically converge on:
> (Amount of RAM used by most expensive request) x (Maximum simultaneous
> Django processes)
Technically it can be worse than that. Absolute worst case is if
multithreaded processes used with multiprocess web server. In that
case:
(Amount of RAM used by most expensive request) x (Maximum
simultaneous Django processes) x (Maximum simultaneous request threads
per process).
You would probably be unlucky to have so many requests for same URL at
same time, but technically possible.
> and nothing I do with within Django itself can reduce this.
>
> Does that sounds about right?
There are few things one can do.
1. Exec a separate process just for purposes of generating the one
PDF. This way the memory usage is at least transient.
2. Have a backend service, communicated to via something like XML-RPC,
which performs the generation of the PDF, with requests potentially
queued up within it so only generating one PDF at a time.
3. Use Apache/mod_wsgi and use multiple daemon process groups, with
the memory consuming URLs delegated to process group of their own,
distinct from all the other URLs in the application. This means that
you can limit how many fat processes there are, make it single
threaded to ensure that maximum memory equates to how much one PDF
takes and not multiple if multiple concurrent requests. Can also set
process to recycle after fewer number of requests than other parts of
the application, or when idle for too long. For example, if your
Django application isn't thread safe anyway:
WSGIScriptAlias / /some/path/django.wsgi
WSGIDaemonProcess main processes=10 threads=1
WSGIDaemonProcess memory-hungry processes=1 threads=1 maximum-
requests=10 inactivity-timeout=30
WSGIProcessGroup main
<Location /memory/hungry/url>
WSGIProcessGroup memory-hungry
</Location>
If your application is thread safe, then could instead have used:
WSGIDaemonProcess main processes=1 threads=10
Graham
> Andy
>
> On Wed, Apr 8, 2009 at 1:31 AM, Malcolm Tredinnick <malc...@pointy-stick.com
>
> > wrote:
>
> > On Tue, 2009-04-07 at 15:20 -0700, andybak wrote:
> > > I've got no really huge tables - the entire db is under 6meg and the
> > > site isn't even public yet so traffic is minimal. Memory just doesn't
> > > seem to go down. (until the process get's restarted by maxchild or by
> > > me killing it).
>
> > That's completely normal behaviour for Unix-like memory management. Once
> > memory is freed, it is returned to the process for later reuse (so a
> > subsequent malloc() call by that process will reuse memory that is had
> > earlier freed). This helps avoid some causes of memory fragmentation and
> > allows for contiguous large blocks to be malloc-ed by other processes.
>
> > It's also precisely why long running processes are often designed to
> > have their memory using portions restart periodically. It's the act of
> > exiting the process at the system level which returns the memory to the
> > system pool.
>
> > Regards,
> > Malcolm
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
