Re: Django Web Application Cost So Many memory

[Graham Dumpleton]

On Thursday, March 31, 2011 4:57:20 PM UTC+11, bird sky wrote:
>
> hi Graham:
>  i'm using the nginx+fcgi . And my startup command is below
>
> python manage.py runfcgi host=127.0.0.1 port=3033 method=prefork 
> pidfile=/var/run/fcgi.pid minspare=5 maxspare=30 maxchildren=60 
> maxrequests=200
>

Same questions I posed before apply here. Why are you using prefork? Why do 
you need to run so many processes? Why can't you use a small number of 
persistent multithreaded processes?

Your memory size for a single process isn't out of the ordinary, it is not 
using multithreading and so that you need so many processes which is going 
to kill you on memory usage.

As with Apache, be wary of starting minimal number of processes and then 
allowing process manager to dynamically increase and decrease number of 
active processes. You will suffer the same load spike problems that you can 
get with Apache as documented in the blog post I linked to.

For fat Python web applications with large start up cost you are better off 
having small fixed number of multithreaded processes. That way memory usage 
is more predictable and you avoid load spikes from spinning up more 
processes. You just need to work out how many fixed processes/threads to use 
to handle realistic level of requests. You can't base it on unrealistic 
loads from benchmarking as in practice you will probably never see such 
levels of requests unless you become very popular very quickly or get 
Slashdotted.
 

> p.s I've invite you on gtalk, hope your response.
>

Talk sessions are almost as bad as IRC for discussing such stuff. Keep it on 
the mailing list.

Graham
 

> On Thu, Mar 31, 2011 at 6:25 AM, Graham Dumpleton 
> wrote:
>
>> Why are you using prefork MPM and running Django embedded that way.
>>
>> Prefork MPM may be fine for PHP, but it is a poor solution for fat Python 
>> web applications unless you are prepared to give it the necessary memory 
>> resources and configure Apache properly specifically for that single Python 
>> web applications requirements.
>>
>> If you are going to use embedded mode of Apache, then you should at least 
>> use worker MPM and then you still need to be mindful of how Apache MPM 
>> settings are configured so as to try and keep processes persistent all the 
>> time and not let Apache kill off/restart processes when it feels it needs to 
>> as the startup load will only make performance worse when you need it, ie., 
>> when load spikes.
>>
>> For a discussion on why embedded mode is not always the best, especially 
>> with prefork MPM, go read:
>>
>>   
>> http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
>>
>> If using Apache/mod_wsgi the preferred setup would be daemon mode where 
>> you can better control processes/threads independent of Apache MPM, meaning 
>> you can still use prefork MPM if you are stuck with support a PHP 
>> application on the same Apache.
>>
>> Graham
>>
>>
>> On Wednesday, March 30, 2011 11:03:37 PM UTC+11, bird sky wrote:
>>>
>>> Hello Everybody: 
>>>  I encounter a problem that my Django project, it has more than 60 
>>> app modules, and some models are very large, more than 30 fields . And 
>>> when I startup my project, regardless of in development server, fast 
>>> cgi(flup),or mod_wsgi. i found it cost at least 60M memory per 
>>> instance. I guess this has some relation with that "django will load 
>>> all models on startup" . but how can I deal with my problem? Because 
>>> if an instance cost 60M memory, when I deploy my project in prefork 
>>> web server with 100 instance, i will cost 6GB memory. I don't think 
>>> this is a normal state.
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To post to this group, send email to django...@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> django-users...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/django-users?hl=en.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[Xavier Ordoquy]

Le 31 mars 2011 à 07:57, 付毅 a écrit :

> hi Graham:
>  i'm using the nginx+fcgi . And my startup command is below
> 
> python manage.py runfcgi host=127.0.0.1 port=3033 method=prefork 
> pidfile=/var/run/fcgi.pid minspare=5 maxspare=30 maxchildren=60 
> maxrequests=200
> 
> p.s I've invite you on gtalk, hope your response.

The thing Graham was pointing out is that you should rather have nginx as a 
frontal for handling connections and acting as a proxy to django applications.
That way Python application are left outside nginx. Nginx deals with 
maintaining connections with the client and Python will only focus on 
processing without being distracted with keeping connections on hold.
This architecture (Nginx + gunicorn) is widely documented, you'll find tons of 
posts about it.

Regards,
Xavier.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[付毅]

hi Graham:
 i'm using the nginx+fcgi . And my startup command is below

python manage.py runfcgi host=127.0.0.1 port=3033 method=prefork
pidfile=/var/run/fcgi.pid minspare=5 maxspare=30 maxchildren=60
maxrequests=200

p.s I've invite you on gtalk, hope your response.

On Thu, Mar 31, 2011 at 6:25 AM, Graham Dumpleton <
graham.dumple...@gmail.com> wrote:

> Why are you using prefork MPM and running Django embedded that way.
>
> Prefork MPM may be fine for PHP, but it is a poor solution for fat Python
> web applications unless you are prepared to give it the necessary memory
> resources and configure Apache properly specifically for that single Python
> web applications requirements.
>
> If you are going to use embedded mode of Apache, then you should at least
> use worker MPM and then you still need to be mindful of how Apache MPM
> settings are configured so as to try and keep processes persistent all the
> time and not let Apache kill off/restart processes when it feels it needs to
> as the startup load will only make performance worse when you need it, ie.,
> when load spikes.
>
> For a discussion on why embedded mode is not always the best, especially
> with prefork MPM, go read:
>
>
> http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
>
> If using Apache/mod_wsgi the preferred setup would be daemon mode where you
> can better control processes/threads independent of Apache MPM, meaning you
> can still use prefork MPM if you are stuck with support a PHP application on
> the same Apache.
>
> Graham
>
>
> On Wednesday, March 30, 2011 11:03:37 PM UTC+11, bird sky wrote:
>>
>> Hello Everybody:
>>  I encounter a problem that my Django project, it has more than 60
>> app modules, and some models are very large, more than 30 fields . And
>> when I startup my project, regardless of in development server, fast
>> cgi(flup),or mod_wsgi. i found it cost at least 60M memory per
>> instance. I guess this has some relation with that "django will load
>> all models on startup" . but how can I deal with my problem? Because
>> if an instance cost 60M memory, when I deploy my project in prefork
>> web server with 100 instance, i will cost 6GB memory. I don't think
>> this is a normal state.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[付毅]

hi Diederik:
I think you can see something from my attach file, that indict i've
60M-70M per python process cost. That is because I start the flup with
command below:
python manager.py runfcgi host=127.0.0.1 port=3033 method=prefork
pidfile=/var/run/fcgi.pid minspare=5 maxspare=30 maxchildren=60
maxrequests=200

it will fork 30 process , and every process will cost nearly the same memory
(60M-70M)

On Thu, Mar 31, 2011 at 5:55 AM, Diederik van der Boor wrote:

> Op woensdag 30 maart 2011 17:48:38 schreef 付毅:
> > hi Xavier Ordoquy
> > So , you mean 60M per python process is in a normal status? I really
> > encouter this kind of website before
>
> Please note this is for the entire website, not per apache instance.
> There is one Django process (via mod_wsgi, or standalone fastcgi) that
> handles all requests coming from visitors.
>
> I hope this was obvious, but I got the impression that bit got lost in the
> discussion,
>
> Greetings,
> Diederik
>
>
> > On Wed, Mar 30, 2011 at 11:35 PM, Xavier Ordoquy
> wrote:
> > > Le 30 mars 2011 à 15:41, 付毅 a écrit :
> > > > i don't think 100 instance means to 100 CPUs. I just want to use
> > >
> > > http_load to test pressure to my web server. if i make the 100
> cocurrent
> > > request per second with 100 cocurrent connection. I will make 100
> python
> > > process instance when I deploy my web project in prefork environment.
> Any
> > > others agree with me?
> > >
> > > If you have 100 instances, 9 of 10 which are waiting in python code,
> > > you've got a huge architecture issue.
> > > Let a front server (nginx, apache, and co) deal with administrative
> tasks
> > > and let django deal with computing the pages.
> > > There are tons of blog posts out there that explains how to do it.
> > >
> > > Regards,
> > > Xavier.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Django users" group.
> > > To post to this group, send email to django-users@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > django-users+unsubscr...@googlegroups.com.
> > > For more options, visit this group at
> > > http://groups.google.com/group/django-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[Graham Dumpleton]

Why are you using prefork MPM and running Django embedded that way.

Prefork MPM may be fine for PHP, but it is a poor solution for fat Python 
web applications unless you are prepared to give it the necessary memory 
resources and configure Apache properly specifically for that single Python 
web applications requirements.

If you are going to use embedded mode of Apache, then you should at least 
use worker MPM and then you still need to be mindful of how Apache MPM 
settings are configured so as to try and keep processes persistent all the 
time and not let Apache kill off/restart processes when it feels it needs to 
as the startup load will only make performance worse when you need it, ie., 
when load spikes.

For a discussion on why embedded mode is not always the best, especially 
with prefork MPM, go read:

  http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

If using Apache/mod_wsgi the preferred setup would be daemon mode where you 
can better control processes/threads independent of Apache MPM, meaning you 
can still use prefork MPM if you are stuck with support a PHP application on 
the same Apache.

Graham

On Wednesday, March 30, 2011 11:03:37 PM UTC+11, bird sky wrote:
>
> Hello Everybody: 
>  I encounter a problem that my Django project, it has more than 60 
> app modules, and some models are very large, more than 30 fields . And 
> when I startup my project, regardless of in development server, fast 
> cgi(flup),or mod_wsgi. i found it cost at least 60M memory per 
> instance. I guess this has some relation with that "django will load 
> all models on startup" . but how can I deal with my problem? Because 
> if an instance cost 60M memory, when I deploy my project in prefork 
> web server with 100 instance, i will cost 6GB memory. I don't think 
> this is a normal state.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[Diederik van der Boor]

Op woensdag 30 maart 2011 17:48:38 schreef 付毅:
> hi Xavier Ordoquy
> So , you mean 60M per python process is in a normal status? I really
> encouter this kind of website before

Please note this is for the entire website, not per apache instance.
There is one Django process (via mod_wsgi, or standalone fastcgi) that
handles all requests coming from visitors.

I hope this was obvious, but I got the impression that bit got lost in the 
discussion,

Greetings,
Diederik

 
> On Wed, Mar 30, 2011 at 11:35 PM, Xavier Ordoquy 
wrote:
> > Le 30 mars 2011 à 15:41, 付毅 a écrit :
> > > i don't think 100 instance means to 100 CPUs. I just want to use
> > 
> > http_load to test pressure to my web server. if i make the 100 cocurrent
> > request per second with 100 cocurrent connection. I will make 100 python
> > process instance when I deploy my web project in prefork environment. Any
> > others agree with me?
> > 
> > If you have 100 instances, 9 of 10 which are waiting in python code,
> > you've got a huge architecture issue.
> > Let a front server (nginx, apache, and co) deal with administrative tasks
> > and let django deal with computing the pages.
> > There are tons of blog posts out there that explains how to do it.
> > 
> > Regards,
> > Xavier.
> > 
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django users" group.
> > To post to this group, send email to django-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > django-users+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[Adam Nelson]

I think 60M is fine.  For us, with nginx in front of gunicorn, we can get 
many simultaneous connections per process.

-Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[Xavier Ordoquy]

Hi,

Sounds correct to me.

Regards,
Xavier.

Le 30 mars 2011 à 17:48, 付毅 a écrit :

> hi Xavier Ordoquy
> So , you mean 60M per python process is in a normal status? I really encouter 
> this kind of website before
> 
> On Wed, Mar 30, 2011 at 11:35 PM, Xavier Ordoquy  wrote:
> 
> Le 30 mars 2011 à 15:41, 付毅 a écrit :
> 
> > i don't think 100 instance means to 100 CPUs. I just want to use http_load 
> > to test pressure to my web server. if i make the 100 cocurrent request per 
> > second with 100 cocurrent connection. I will make 100 python process 
> > instance when I deploy my web project in prefork environment. Any others 
> > agree with me?
> 
> If you have 100 instances, 9 of 10 which are waiting in python code, you've 
> got a huge architecture issue.
> Let a front server (nginx, apache, and co) deal with administrative tasks and 
> let django deal with computing the pages.
> There are tons of blog posts out there that explains how to do it.
> 
> Regards,
> Xavier.
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-users?hl=en.
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[付毅]

hi Xavier Ordoquy
So , you mean 60M per python process is in a normal status? I really
encouter this kind of website before

On Wed, Mar 30, 2011 at 11:35 PM, Xavier Ordoquy wrote:

>
> Le 30 mars 2011 à 15:41, 付毅 a écrit :
>
> > i don't think 100 instance means to 100 CPUs. I just want to use
> http_load to test pressure to my web server. if i make the 100 cocurrent
> request per second with 100 cocurrent connection. I will make 100 python
> process instance when I deploy my web project in prefork environment. Any
> others agree with me?
>
> If you have 100 instances, 9 of 10 which are waiting in python code, you've
> got a huge architecture issue.
> Let a front server (nginx, apache, and co) deal with administrative tasks
> and let django deal with computing the pages.
> There are tons of blog posts out there that explains how to do it.
>
> Regards,
> Xavier.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[Xavier Ordoquy]

Le 30 mars 2011 à 15:41, 付毅 a écrit :

> i don't think 100 instance means to 100 CPUs. I just want to use http_load to 
> test pressure to my web server. if i make the 100 cocurrent request per 
> second with 100 cocurrent connection. I will make 100 python process instance 
> when I deploy my web project in prefork environment. Any others agree with me?

If you have 100 instances, 9 of 10 which are waiting in python code, you've got 
a huge architecture issue.
Let a front server (nginx, apache, and co) deal with administrative tasks and 
let django deal with computing the pages.
There are tons of blog posts out there that explains how to do it.

Regards,
Xavier.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[付毅]

hi Bennett:
i attache my ps output file, just see the RSS column, that is the real
memory usage for every instance


On Wed, Mar 30, 2011 at 9:51 PM, James Bennett wrote:

> On Wed, Mar 30, 2011 at 8:41 AM, 付毅  wrote:
> > i don't think 100 instance means to 100 CPUs. I just want to use
> http_load
> > to test pressure to my web server. if i make the 100 cocurrent request
> per
> > second with 100 cocurrent connection. I will make 100 python process
> > instance when I deploy my web project in prefork environment. Any others
> > agree with me?
>
> Well, I'm not sure I agree with how you're interpreting your stats; a
> process can report that it's using 60MB, but that doesn't mean that
> two such processes would consume a total of 120MB of memory.
>
> I'd also be curious to know what sort of stuff you're doing that's
> using (allegedly) so much memory per-process.
>
>
> --
> "Bureaucrat Conrad, you are technically correct -- the best kind of
> correct."
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

USER   PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED  TIME COMMAND
root 78171 11.4  0.8 128408 63480  ??  S 9:58下午   0:03.86 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78113  2.9  0.8 139344 70208  ??  S 9:35下午   0:20.62 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78094  0.6  0.8 139344 70464  ??  S 9:30下午   0:22.42 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78152  0.5  0.8 128408 63484  ??  S 9:50下午   0:07.20 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78112  0.4  0.8 139344 70200  ??  S 9:35下午   0:15.93 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78193  0.4  0.7 126360 60916  ??  S10:03下午   0:00.17 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78170  0.3  0.8 137296 68816  ??  S 9:58下午   0:03.78 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78124  0.2  0.8 139344 70256  ??  S 9:43下午   0:15.04 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78154  0.2  0.8 137296 68748  ??  S 9:54下午   0:05.87 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78148  0.1  0.8 139344 70188  ??  S 9:48下午   0:10.54 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78192  0.1  0.7 126360 61760  ??  S10:01下午   0:00.95 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78088  0.0  0.8 139344 70508  ??  S 9:28下午   0:25.50 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78089  0.0  0.8 139344 70184  ??  S 9:29下午   0:22.32 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78090  0.0  0.8 139344 70236  ??  S 9:29下午   0:19.92 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78091  0.0  0.8 139344 70528  ??  S 9:30下午   0:21.93 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78096  0.0  0.8 139344 70476  ??  S 9:30下午   0:18.26 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78097  0.0  0.8 139344 70260  ??  S 9:30下午   0:24.18 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78114  0.0  0.8 139344 70148  ??  S 9:35下午   0:15.27 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78115  0.0  0.8 139344 70212  ??  S 9:35下午   0:19.70 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78116  0.0  0.8 139344 70220  ??  S 9:36下午   0:13.64 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78118  0.0  0.8 139344 70216  ??  S 9:37下午   0:14.99 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78119  0.0  0.8 139344 70248  ??  S 9:39下午   0:15.21 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78123  0.0  0.8 139344 70180  ??  S 9:40下午   0:11.17 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78125  0.0  0.8 139344 70208  ??  S 9:43下午   0:09.74 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78142  0.0  0.8 139344 70176  ??  S 9:45下午   0:15.32 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78143  0.0  0.8 139344 70192  ??  S 9:46下午   0:11.42 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78153  0.0  0.8 139344 70148  ??  S 9:52下午   0:10.31 python 
/home/x/www/capitalog/manage.py runfcgi h
root 78169  0.0  0.8 139344 70088  ??  S 9:58下午   0:05.28 

Re: Django Web Application Cost So Many memory

[James Bennett]

On Wed, Mar 30, 2011 at 8:41 AM, 付毅  wrote:
> i don't think 100 instance means to 100 CPUs. I just want to use http_load
> to test pressure to my web server. if i make the 100 cocurrent request per
> second with 100 cocurrent connection. I will make 100 python process
> instance when I deploy my web project in prefork environment. Any others
> agree with me?

Well, I'm not sure I agree with how you're interpreting your stats; a
process can report that it's using 60MB, but that doesn't mean that
two such processes would consume a total of 120MB of memory.

I'd also be curious to know what sort of stuff you're doing that's
using (allegedly) so much memory per-process.


-- 
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[付毅]

I've found one of my app module has a large model. there are 207 models in
models.py.  When I add this app module into INSTALL_APPS, the process
instance memory increase nearly 10M more.

I think can I move these model classes out of my models.py file and move
them in to another .py file. And when I want to use it , just Dynamic Load
them as need?? Any suggestions?

On Wed, Mar 30, 2011 at 9:41 PM, 付毅  wrote:

> i don't think 100 instance means to 100 CPUs. I just want to use http_load
> to test pressure to my web server. if i make the 100 cocurrent request per
> second with 100 cocurrent connection. I will make 100 python process
> instance when I deploy my web project in prefork environment. Any others
> agree with me?
>
>
> On Wed, Mar 30, 2011 at 9:07 PM, Xavier Ordoquy wrote:
>
>> Hi,
>>
>> > Because
>> > if an instance cost 60M memory, when I deploy my project in prefork
>> > web server with 100 instance, i will cost 6GB memory. I don't think
>> > this is a normal state.
>>
>> I hardly see the need for 100 instances.
>> Could you elaborate on that need ?
>> Imagine those 100 instances are processing at full speed. You'll need 100
>> CPUs, not counting databases.
>> I doubt 6Gb would be an issue if you already have 100 CPUs.
>>
>> Regards,
>> Xavier.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To post to this group, send email to django-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> django-users+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/django-users?hl=en.
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[付毅]

i don't think 100 instance means to 100 CPUs. I just want to use http_load
to test pressure to my web server. if i make the 100 cocurrent request per
second with 100 cocurrent connection. I will make 100 python process
instance when I deploy my web project in prefork environment. Any others
agree with me?

On Wed, Mar 30, 2011 at 9:07 PM, Xavier Ordoquy wrote:

> Hi,
>
> > Because
> > if an instance cost 60M memory, when I deploy my project in prefork
> > web server with 100 instance, i will cost 6GB memory. I don't think
> > this is a normal state.
>
> I hardly see the need for 100 instances.
> Could you elaborate on that need ?
> Imagine those 100 instances are processing at full speed. You'll need 100
> CPUs, not counting databases.
> I doubt 6Gb would be an issue if you already have 100 CPUs.
>
> Regards,
> Xavier.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[Xavier Ordoquy]

Hi,

> Because
> if an instance cost 60M memory, when I deploy my project in prefork
> web server with 100 instance, i will cost 6GB memory. I don't think
> this is a normal state.

I hardly see the need for 100 instances.
Could you elaborate on that need ?
Imagine those 100 instances are processing at full speed. You'll need 100 CPUs, 
not counting databases.
I doubt 6Gb would be an issue if you already have 100 CPUs.

Regards,
Xavier.


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[付毅]

no. i don't use any cache, and I 've try to remove the app module which I
doubt cost memory. I found that the most memory eater is the app module who
has very large models.py

On Wed, Mar 30, 2011 at 8:32 PM, moham...@efazati.org
wrote:

>  do you have any cache ing?
>
> On 03/30/2011 04:33 PM, bird sky wrote:
>
> Hello Everybody:
>  I encounter a problem that my Django project, it has more than 60
> app modules, and some models are very large, more than 30 fields . And
> when I startup my project, regardless of in development server, fast
> cgi(flup),or mod_wsgi. i found it cost at least 60M memory per
> instance. I guess this has some relation with that "django will load
> all models on startup" . but how can I deal with my problem? Because
> if an instance cost 60M memory, when I deploy my project in prefork
> web server with 100 instance, i will cost 6GB memory. I don't think
> this is a normal state.
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Web Application Cost So Many memory

[moham...@efazati.org]

do you have any cache ing?

On 03/30/2011 04:33 PM, bird sky wrote:

Hello Everybody:
  I encounter a problem that my Django project, it has more than 60
app modules, and some models are very large, more than 30 fields . And
when I startup my project, regardless of in development server, fast
cgi(flup),or mod_wsgi. i found it cost at least 60M memory per
instance. I guess this has some relation with that "django will load
all models on startup" . but how can I deal with my problem? Because
if an instance cost 60M memory, when I deploy my project in prefork
web server with 100 instance, i will cost 6GB memory. I don't think
this is a normal state.



--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.