Can't you use crontab to submit your conversion task?
Cheers,
Alan
On Sat, Jan 10, 2009 at 09:19, Stefan Tunsch <stun...@gmail.com> wrote:
>
> It looks like it's clear video conversion is your bottleneck.
>
> I would also suggest doing that on a different worker process.
>
> You will also want to avoid converting the same file again if you've
> done it before...
>
> I was suggesting to give a shot of using ONLY nginx with Django.
> If you search in Google you'll find docs that help you setting that up.
>
> In general, 1GB of RAM is not much for a site that coverts video on the
> fly...
>
> Regards
>
> On 1/10/09, Graham Dumpleton <graham.dumple...@gmail.com> wrote:
> >
> >
> >
> > On Jan 10, 11:05 am, lenin <lenin....@gmail.com> wrote:
> >> The server MPM is prefork.
> >>
> >> I had mod_wsgi in daemon mode I think, with this lines:
> >>
> >> WSGIDaemonProcess user_name user=user_name group=user_name
> >> WSGIProcessGroup user_name
> >> WSGIScriptAlias / /path/to/django.wsgi
> >
> > Which is one process with 15 threads. That means at most 15 requests
> > can be handled at the same time.
> >
> > The problem is that with what you are doing, is it is likely that the
> > requests take a long time if they doing video conversion. Thus if you
> > get 15 requests active at one time, other requests will be queued and
> > will appear to take a lot longer.
> >
> > How long does a request which involves conversion normally take?
> > Knowing this is critical to determining how many processes/threads you
> > need to handle enough concurrent requests based on your load.
> >
> > The next problem though is if you are doing the video conversion
> > within the Django process itself, then allowing more concurrent
> > requests and thus conversions, means that the process size will
> > balloon out if this process takes a lot of memory for each request.
> > So, are you doing conversion within the Django process and how much
> > memory does the process of conversion use.
> >
> > You thus perhaps should not be doing video conversion within the same
> > process, but executing a separate application to perform the
> > conversion. Even then, you may need a queueing system or otherwise
> > some way of restricting how many conversions can occur at a time,
> > because even though performed by a separate application, it may still
> > take up a lot of transient memory while running.
> >
> > Also, from what I understand Django's abilities to serve up static
> > files or streamed data, isn't that excellent. In particular, if
> > hosting with WSGI capable server, it doesn't use the wsgi.file_wrapper
> > extension. This extension ensures that files are streamed in blocks
> > and the file is not read all into memory first in order to be able to
> > send it. Also, if operating system provides it, Apache will use
> > sendfile() or memory mapping techniques to return data in a much more
> > efficient way. So, how exactly are you returning the converted data
> > for Django to respond with?
> >
> >> That was at the moment of the first message, but now I have commented
> >> out the WSGIDaemonProcess and WSGIProcessGroup directives, and what
> >> happens is that the memory fills up until the server crashes, It runs
> >> out of RAM and swap and it just dies. But until it crashes it is very
> >> responsive.
> >
> > Which is not surprising as prefork means that instead of a restricted
> > number of copies of Apache, based on your configuration you could have
> > up to 70. That is 70 copies of Django, plus memory from conversions.
> > This would quickly cripple a box without adequate memory. Just the
> > startup cost of the extra processes when Apache deems they need to be
> > created would cause the load average of the box to shoot up.
> >
> >> I have tried putting processes=5 threads=1 maximum-
> >> requests=200 in the daemon configuration but it doesn't help.
> >
> > Which is only 5 concurrent requests, which is worse than the 15 you
> > had before. You also have 5 potentially fat Django processes rather
> > than 1.
> >
> > Graham
> >
> >> On apache I have this configuration
> >> KeepAlive Off
> >> <IfModule mpm_prefork_module>
> >> StartServers 5
> >> MinSpareServers 5
> >> MaxSpareServers 10
> >> MaxClients 70
> >> MaxRequestsPerChild 150
> >> </IfModule>
> >>
> >> On nginx I have
> >> worker_processes 10;
> >> events {
> >> worker_connections 4096;
> >> use epoll;}
> >>
> >> location / {
> >> proxy_pass http://localhost:8080;
> >> proxy_redirect default;
> >> proxy_set_header Host $host;
> >> proxy_set_header X-Real-IP $remote_addr;
> >> proxy_set_header X-Forwarded-For
> >> $proxy_add_x_forwarded_for;
> >>
> >> client_max_body_size 10m;
> >> client_body_buffer_size 128k;
> >>
> >> proxy_connect_timeout 1000;
> >> proxy_send_timeout 1000;
> >> proxy_read_timeout 1000;
> >>
> >> proxy_buffer_size 4k;
> >> proxy_buffers 4 32k;
> >> proxy_busy_buffers_size 64k;
> >> proxy_temp_file_write_size 128k;
> >> }
> >> I only include the bits I think are relevant.
> >>
> >> The site is kickyoutube.com, you can download any youtube video in
> >> several formats by appending kick to the youtube video url. Although
> >> at this moment I have disabled the formats that need conversion.
> >>
> >> I'm using mysql for the database and it is on the same server, but it
> >> really doesn't do a lot of queries.
> >>
> >> I have memcached but I only cache the list of latest videos at the
> >> homepage.
> >>
> >> So it really is the video conversion what takes up all the memory,
> >> because when I disable it everything seems normal.
> >>
> >> On 9 ene, 15:06, Graham Dumpleton <graham.dumple...@gmail.com> wrote:
> >>
> >> > On Jan 10, 4:21 am, lenin <lenin....@gmail.com> wrote:
> >>
> >> > > Hello,
> >>
> >> > > I would like to know if you could help me solve an issue with my
> >> > > server setup. I have a Django application running with mod_wsgi on
> >> > > apache with nginx as frontend, and it got very popular so the server
> >> > > gets more than 10000 requests a day. The app does some video
> >> > > conversion with ffmpeg and mencoder, using the commands module to
> call
> >> > > the commands.
> >>
> >> > > The server has 1 GB of RAM and an Intel Xeon 3060 processor.
> >>
> >> > > At first the server would crash and I needed to reboot it, and the
> >> > > problem seemed to be that the system ran out of swap, so I thought
> it
> >> > > was a memory issue, did some modifications and now memory usage is
> >> > > stable.
> >>
> >> > > But now the problem is that the site takes too much to load some
> >> > > times, I think because of all the simultaneous requests. I was
> >> > > wondering what would be the optimal apache+mod_wsgi+nginx
> >> > > configuration for a setup like this.
> >>
> >> > > Any help will be very much appreciated.
> >>
> >> > What MPM is Apache running with, prefork or worker MPM? Run 'httpd -V'
> >> > if you don't know.
> >>
> >> > What is the mod_wsgi configuration you are using for setting up your
> >> > Django instance? Are you using embedded mode or daemon mode? Post
> >> > actual WSGI configuration bits, do not describe it.
> >>
> >> > BTW, 10000 requests a day is not much. ;-)
> >>
> >> > Graham
> > >
> >
>
> >
>
--
Alan Wilter S. da Silva, D.Sc. - CCPN Research Associate
Department of Biochemistry, University of Cambridge.
80 Tennis Court Road, Cambridge CB2 1GA, UK.
>>http://www.bio.cam.ac.uk/~awd28<<
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
