Re: Memory limits
It also depends on how OP deploys Django. Use embedding with mod_python or mod_wsgi, instead of daemon mode of mod_wsgi or fastcgi, then you can be setting yourself up for problems. Read: http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html Graham On Sep 14, 1:52 am, Peter Bengtsson wrote: > Hard to say as it depends on your app but can't you just run the app > on your laptop and see how much memory it takes up when you run some > basic stresstests. > Django is quite close to pure python but when you extract large lists > of model instance objects into lists it can push the memory > consumption. > > On Sep 13, 1:14 pm, Pablo Escobar wrote: > > > > > Hi > > i'm wondering what is memory consumption for django 1.1 + postgre. > > Will it be enough to have VPS with 256 MB of RAM as entry level for > > recent projects? > > > thanks --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Memory limits
Hard to say as it depends on your app but can't you just run the app on your laptop and see how much memory it takes up when you run some basic stresstests. Django is quite close to pure python but when you extract large lists of model instance objects into lists it can push the memory consumption. On Sep 13, 1:14 pm, Pablo Escobar wrote: > Hi > i'm wondering what is memory consumption for django 1.1 + postgre. > Will it be enough to have VPS with 256 MB of RAM as entry level for > recent projects? > > thanks --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Memory limits
Hi i'm wondering what is memory consumption for django 1.1 + postgre. Will it be enough to have VPS with 256 MB of RAM as entry level for recent projects? thanks --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
> > > BTW, if you run 'ldd' on the mod_python.so file from the Apache > > > modules directory, does it use Python as a shared library or is there > > > no reference to libpython2.?.so at all, meaning it is embedded with in > > > mod_python.so? What is the actual size of your mod_python.so file? > > > mod_python.so is about 4MB. It doesn't show libpython2.x.so as one of > > the shared libraries. > > Which shows that the Python installation on the box is really crappy. > One of the problems with using RedHat derived distributions. > > That the mod_python.so is so large means that there is no shared > library built and provided with the Python installation, or at least > not with the version that mod_python was compiled against. > Alternatively, there is a shared library, but it is only in /usr/lib > and not along side the libpython2.?.a file in /usr/lib/python2.?/ > config directory. > > In other words, when mod_python was built, all it found in the /usr/ > lib/python.2?/config directory was the static library archive. As a > consequence, 'libtool' when it creates the mod_python.so file has no > choice but to extract all the object files from the static Python > library and embed them into mod_python.so itself. Thus, instead of > mod_python.so being less than 0.5MB through being able to link > dynamically to the Python library it is much larger. > > Worse is that the size of mod_python.so indicates that debug may have > been compiled into the Python library. For an optimised build of the > Python library it is normally only about 1MB, not over 3MB as in this > case. > > Even more worse than that, is what now happens when that mod_python.so > is loaded by Apache. Now the way I understand it, normally a > dynamically loaded module should really be shared and thus memory cost > only applies once across all processes that load it, however, if the > object files in that static Python library are not truly position > independent, when mod_python.so gets loaded, the dynamic linker will > have to perform address relocations to fixup references to function > and data from the Python library objects. As a consequence, rather > than being shared memory, the mod_python.so becomes private memory to > the parent Apache process. When Apache forks off the child processes, > they in turn will each end up with their own private copy of up to 4MB > of memory that mod_python.so consumes when it could all have been > shared. So, even on a fresh restart, the Apache processes are going to > be much larger than they need to be because of the way that Python was > built in the first place. > > Now whether this is going to happen like this I am not 100% sure as it > can depend on the operating system and perhaps the age of the > operating system version as well and how they handle loading of > dynamically loadable modules. I am led to believe that this can occur > on Solaris, to what degree it applies on different Linux versions I am > not as sure. > > Either way, that debug seems to be built into your Python library is > not good. IfWebfactionused a better build of Python with > optimisation and a shared library they could quite easily cut down on > memory use allowing your applications to have more space within their > set limit. > > What is even more annoying is that various people run around > continually saying how crap mod_python is because of how much memory > it uses, when in practice a lot of it has to do with poorly built > Python versions as I show above. :-( > > > > Base Django setup can take up 5-7MB without even your own stuff so it > > > can jump up quite a bit. Is that 35MB in total across all processes or > > > per process? > > > About 4MB for the parent process, and 13MB for the two child > > processes. > > > > What version of mod_python are you using? > > > 3.2.8. > > Which has various memory leaks. Even 3.2.10 has memory leaks. I highly > recommend upgrading to mod_python 3.3.1 if you can do it as it > eliminates a lot of memory leaks. If you also do a rebuild of Python > with optimisation and a shared library you should hopefully see less > memory being used then at startup. Just a quick update to mention that we've taken these comments into account and we've optimized our Django setup at WebFaction: - Django apps now comes with mod_python-3.3.1 by default - mod_python is now compiled with a shared libpython.so (the size of mod_python.so went from 3MB down to 300KB). - you can now choose between Python-2.4 or Python-2.5 when creating a Django app. Remi -- WebFaction - Hosting for an agile web http://www.webfaction.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~
Re: Django, shared hosting memory limits and memory optimization
That's a bit creepy. If not the DEBUG issue, then what? The parent Apache process hasn't budged from 3MB since the restart and I can't imagine what else would have changed between the 90MB period and the 30MB period. I loaded my project from svn so there were no .pyc files initially, I've never used the built-in dev server, and I didn't delete the .pyc files when I restarted Apache so I doubt that's made any difference. On Jun 2, 6:41 pm, Graham Dumpleton <[EMAIL PROTECTED]> wrote: > The problem with that hypothesis is that mod_python in the Apache > parent process never actually imports any of your user code modules > for Django, that is only ever done in the child processes. Thus, there > is no way it could have remembered the setting in the first place. > This is why I couldn't understand why your Apache parent process was > so big, as it effectively doesn't do that much. > > One possibility for child processes being so big is that for some > reason Python was loading the .pyc file for the settings file and it > was produced from settings.py when DEBUG=True was set but for some > reason when it was set to DEBUG=False the .pyc file wasn't regenerated > and Python was still using it for some reason. > > Thus, as a precaution, it may be advisable to delete all .pyc files if > you have run the development Django server and then subsequently using > Apache on the same files. This would especially be required if you had > copied all the .py/.pyc files from another box and not preserved date/ > time stamps as the copy would have set the .py/.pyc files to be the > same and thus Python wouldn't realise the .py file was actually newer. > Do note that although Python running as used by mod_python will > use .pyc files, it will not replace them if Apache runs as a user > different to the owner of the directories/files. > > Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
BTW, one further thing you can experiment with as far as trying to bring down memory use, is if you have access to main Apache configuration file, try setting: PythonOptimize On at global scope outside of any virtual host containers. This will have the same affect as having supplied '-O -O' option to command line Python. Do note though that this will result in doc strings being discarded, which is where potential for memory savings come from, but some Python modules rely on doc strings to be present to work. Thus, you yo will need to do sufficient testing to ensure you aren't using any Python modules which have a dependency on doc strings being present. Graham On Jun 3, 8:41 am, Graham Dumpleton <[EMAIL PROTECTED]> wrote: > On Jun 3, 5:40 am, chrominance <[EMAIL PROTECTED]> wrote: > > > Epilogue: it looks like the major culprit behind my skyrocketing > > memory usage was indeed my failure to properly restart all the apache > > processes; it looks like the parent process was still storing debug > > info from when I had DEBUG=True in my settings.py, so that setting it > > to False and soft restarting via apachectl had no effect. For the past > > couple of days memory has been sitting between 30-35MB, which is a > > hell of a lot better. > > The problem with that hypothesis is that mod_python in the Apache > parent process never actually imports any of your user code modules > for Django, that is only ever done in the child processes. Thus, there > is no way it could have remembered the setting in the first place. > This is why I couldn't understand why your Apache parent process was > so big, as it effectively doesn't do that much. > > One possibility for child processes being so big is that for some > reason Python was loading the .pyc file for the settings file and it > was produced from settings.py when DEBUG=True was set but for some > reason when it was set to DEBUG=False the .pyc file wasn't regenerated > and Python was still using it for some reason. > > Thus, as a precaution, it may be advisable to delete all .pyc files if > you have run the development Django server and then subsequently using > Apache on the same files. This would especially be required if you had > copied all the .py/.pyc files from another box and not preserved date/ > time stamps as the copy would have set the .py/.pyc files to be the > same and thus Python wouldn't realise the .py file was actually newer. > Do note that although Python running as used by mod_python will > use .pyc files, it will not replace them if Apache runs as a user > different to the owner of the directories/files. > > Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
On Jun 3, 5:40 am, chrominance <[EMAIL PROTECTED]> wrote: > Epilogue: it looks like the major culprit behind my skyrocketing > memory usage was indeed my failure to properly restart all the apache > processes; it looks like the parent process was still storing debug > info from when I had DEBUG=True in my settings.py, so that setting it > to False and soft restarting via apachectl had no effect. For the past > couple of days memory has been sitting between 30-35MB, which is a > hell of a lot better. The problem with that hypothesis is that mod_python in the Apache parent process never actually imports any of your user code modules for Django, that is only ever done in the child processes. Thus, there is no way it could have remembered the setting in the first place. This is why I couldn't understand why your Apache parent process was so big, as it effectively doesn't do that much. One possibility for child processes being so big is that for some reason Python was loading the .pyc file for the settings file and it was produced from settings.py when DEBUG=True was set but for some reason when it was set to DEBUG=False the .pyc file wasn't regenerated and Python was still using it for some reason. Thus, as a precaution, it may be advisable to delete all .pyc files if you have run the development Django server and then subsequently using Apache on the same files. This would especially be required if you had copied all the .py/.pyc files from another box and not preserved date/ time stamps as the copy would have set the .py/.pyc files to be the same and thus Python wouldn't realise the .py file was actually newer. Do note that although Python running as used by mod_python will use .pyc files, it will not replace them if Apache runs as a user different to the owner of the directories/files. Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
Epilogue: it looks like the major culprit behind my skyrocketing memory usage was indeed my failure to properly restart all the apache processes; it looks like the parent process was still storing debug info from when I had DEBUG=True in my settings.py, so that setting it to False and soft restarting via apachectl had no effect. For the past couple of days memory has been sitting between 30-35MB, which is a hell of a lot better. While I was looking up memory optimization stuff I stumbled upon a couple of useful links I'll throw out for posterity's sake: http://www.car-chase.net/2007/apr/30/django-deployment-lessons-learned/ http://emergent.urbanpug.com/?p=60 http://blog.webfaction.com/tips-to-keep-your-django-mod-python-memory-usage-down A couple of people pointed the Webfaction crew to Graham's comments and they say they're looking into the mod_python setup, so hopefully that leads to some results down the road. They've also been testing Litespeed/FastCGI setups, which might help with limiting memory usage via FastCGI config. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
On May 30, 1:52 pm, chrominance <[EMAIL PROTECTED]> wrote: > > BTW, if you run 'ldd' on the mod_python.so file from the Apache > > modules directory, does it use Python as a shared library or is there > > no reference to libpython2.?.so at all, meaning it is embedded with in > > mod_python.so? What is the actual size of your mod_python.so file? > > mod_python.so is about 4MB. It doesn't show libpython2.x.so as one of > the shared libraries. Which shows that the Python installation on the box is really crappy. One of the problems with using RedHat derived distributions. That the mod_python.so is so large means that there is no shared library built and provided with the Python installation, or at least not with the version that mod_python was compiled against. Alternatively, there is a shared library, but it is only in /usr/lib and not along side the libpython2.?.a file in /usr/lib/python2.?/ config directory. In other words, when mod_python was built, all it found in the /usr/ lib/python.2?/config directory was the static library archive. As a consequence, 'libtool' when it creates the mod_python.so file has no choice but to extract all the object files from the static Python library and embed them into mod_python.so itself. Thus, instead of mod_python.so being less than 0.5MB through being able to link dynamically to the Python library it is much larger. Worse is that the size of mod_python.so indicates that debug may have been compiled into the Python library. For an optimised build of the Python library it is normally only about 1MB, not over 3MB as in this case. Even more worse than that, is what now happens when that mod_python.so is loaded by Apache. Now the way I understand it, normally a dynamically loaded module should really be shared and thus memory cost only applies once across all processes that load it, however, if the object files in that static Python library are not truly position independent, when mod_python.so gets loaded, the dynamic linker will have to perform address relocations to fixup references to function and data from the Python library objects. As a consequence, rather than being shared memory, the mod_python.so becomes private memory to the parent Apache process. When Apache forks off the child processes, they in turn will each end up with their own private copy of up to 4MB of memory that mod_python.so consumes when it could all have been shared. So, even on a fresh restart, the Apache processes are going to be much larger than they need to be because of the way that Python was built in the first place. Now whether this is going to happen like this I am not 100% sure as it can depend on the operating system and perhaps the age of the operating system version as well and how they handle loading of dynamically loadable modules. I am led to believe that this can occur on Solaris, to what degree it applies on different Linux versions I am not as sure. Either way, that debug seems to be built into your Python library is not good. If Webfaction used a better build of Python with optimisation and a shared library they could quite easily cut down on memory use allowing your applications to have more space within their set limit. What is even more annoying is that various people run around continually saying how crap mod_python is because of how much memory it uses, when in practice a lot of it has to do with poorly built Python versions as I show above. :-( > > Base Django setup can take up 5-7MB without even your own stuff so it > > can jump up quite a bit. Is that 35MB in total across all processes or > > per process? > > About 4MB for the parent process, and 13MB for the two child > processes. > > > What version of mod_python are you using? > > 3.2.8. Which has various memory leaks. Even 3.2.10 has memory leaks. I highly recommend upgrading to mod_python 3.3.1 if you can do it as it eliminates a lot of memory leaks. If you also do a rebuild of Python with optimisation and a shared library you should hopefully see less memory being used then at startup. Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
> Depends on what RSS means for that platform when running ps. This may > count private memory used plus shared memory use. If Webfaction is > counting shared memory use in your 40MB limit would suck somewhat, as > they would be double counting across all processes. > > If you run 'top' it generally splits resident (private) from shared in > what it outputs. You may be able to get 'ps' to show them separately, > but need to find which field to display. I tried top initially but I couldn't get it to return private memory at all. I'm not sure if that's just a quirk of Red Hat or a quirk of Webfaction's shared hosting setup. > BTW, if you run 'ldd' on the mod_python.so file from the Apache > modules directory, does it use Python as a shared library or is there > no reference to libpython2.?.so at all, meaning it is embedded with in > mod_python.so? What is the actual size of your mod_python.so file? > mod_python.so is about 4MB. It doesn't show libpython2.x.so as one of the shared libraries. > Base Django setup can take up 5-7MB without even your own stuff so it > can jump up quite a bit. Is that 35MB in total across all processes or > per process? > About 4MB for the parent process, and 13MB for the two child processes. > What version of mod_python are you using? 3.2.8. It's been a couple of hours since the hard restart and memory usage is still at about 35MB. I'm now wondering if perhaps the last time I did a full restart (if ever), I'd left Debug = True in the settings.py, and the SQL query storage hadn't been emptied or some other junk. Perhaps doing the full stop/start with Debug = False set might've cleared out all the garbage. I'll watch it over the next couple of hours with my fingers crossed... --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
2 429 0 31 031136848 - S ?? > > 0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start > >70 433 429 0 31 045296 1172 - S ?? > > 0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start > >70 434 429 0 31 045296 1176 - S ?? > > 0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start > > > The parent process here is 429. Normally this process should only use > > quite small amounts of private memory. I have never seen an instance > > of where the parent process ballooned out in size, but if you have a > > dodgy Apache module it feasibly might. For example, from 'top': > > > PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE > > VSIZE > > 434 httpd0.0% 0:00.01 275593 496K 2.73M 1.15M > > 44.2M > > 433 httpd0.0% 0:00.01 275593 496K 2.73M 1.14M > > 44.2M > > 432 httpd0.0% 0:00.00 11037 204K 2.50M 848K > > 30.4M > > 431 httpd0.0% 0:00.00 11037 208K 2.50M 852K > > 30.4M > > 430 httpd0.0% 0:00.00 11033 188K 2.02M 592K > > 29.8M > > 429 httpd0.0% 0:00.11 1113860K 2.73M 1.96M > > 30.6M > > > Ie., 60Kbytes here only, versus 200-500Kbytes for children. > > > Anyway, work out whether how you are restarting Apache is killing of > > the parent process. If not work out how to restart Apache so it is. > > Also look at how much memory the parent is using and see if it is > > small. > > > This at least is some sanity checks you can do. > > > BTW, what Apache modules are actually being loaded using LoadModule > > directive. > > > Graham > > > On May 30, 11:42 am, chrominance <[EMAIL PROTECTED]> wrote: > > > > I've restarted apache several times, using the restart command in the > > > webapps/django directory provided by Webfaction. Perhaps there's a > > > more absolute restart command I don't know about, but I'm pretty sure > > > that's the one--whenever I use it, memory usage seems to go down. > > > > Also, I'm not sure if this is relevant, but whenever I do restart the > > > apache processes, memory usage doesn't reset to the same baseline > > > level--for example, the site ran just fine with about 30-40MB of RAM > > > over the weekend, but restarting apache now only gets us down to about > > > 65MB. And upon my last check the apache processes peaked at about > > > 85MB. If there's a memory leak in my application, it wouldn't still > > > leak after I've restarted apache, would it? > > > > I'll try the Webfaction forum too. Thanks, and keep the advice coming! > > > > On May 29, 8:15 pm, "Jay Parlar" <[EMAIL PROTECTED]> wrote: > > > > > On 5/29/07, chrominance <[EMAIL PROTECTED]> wrote: > > > > > > I've recently put up a newspaper site on Webfaction that was developed > > > > > without much concern for memory limits--coming from PHP, my knowledge > > > > > of memory issues is practically nil. Of course, Webfaction's plans all > > > > > have memory limits, and we're currently on Shared 1, which imposes a > > > > > 40MB limit. We're running the standard apache2/mod_pythonconfig, with > > > > > media served via a separate apache2 process as Webfaction and the > > > > > Django site recommends. The Debug setting has been set to False. I > > > > > believe those are the standard steps recommended for reducing resource > > > > > usage. > > > > > Did you remember to restart Apache after setting Debug to False? I > > > > have a site running on Shared1, and the memory usage was close to what > > > > you have, until I set Debug to False and restarted. > > > > > Also, you might want to ping the Webfaction Django forum > > > > (http://forum.webfaction.com/viewforum.php?id=19) as they're pretty > > > > good about responding, and one of the admins might check if something > > > > is wrong with your config. > > > > > And I *believe* the "3 small sites" comes from the fact that on > > > > Shared1, a max of 3 Apache processes will run, so you can run a site > > > > on each process. I might be completely wrong on that though. > > > > > Jay P --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
ided by Webfaction. Perhaps there's a > > more absolute restart command I don't know about, but I'm pretty sure > > that's the one--whenever I use it, memory usage seems to go down. > > > Also, I'm not sure if this is relevant, but whenever I do restart the > > apache processes, memory usage doesn't reset to the same baseline > > level--for example, the site ran just fine with about 30-40MB of RAM > > over the weekend, but restarting apache now only gets us down to about > > 65MB. And upon my last check the apache processes peaked at about > > 85MB. If there's a memory leak in my application, it wouldn't still > > leak after I've restarted apache, would it? > > > I'll try the Webfaction forum too. Thanks, and keep the advice coming! > > > On May 29, 8:15 pm, "Jay Parlar" <[EMAIL PROTECTED]> wrote: > > > > On 5/29/07, chrominance <[EMAIL PROTECTED]> wrote: > > > > > I've recently put up a newspaper site on Webfaction that was developed > > > > without much concern for memory limits--coming from PHP, my knowledge > > > > of memory issues is practically nil. Of course, Webfaction's plans all > > > > have memory limits, and we're currently on Shared 1, which imposes a > > > > 40MB limit. We're running the standard apache2/mod_pythonconfig, with > > > > media served via a separate apache2 process as Webfaction and the > > > > Django site recommends. The Debug setting has been set to False. I > > > > believe those are the standard steps recommended for reducing resource > > > > usage. > > > > Did you remember to restart Apache after setting Debug to False? I > > > have a site running on Shared1, and the memory usage was close to what > > > you have, until I set Debug to False and restarted. > > > > Also, you might want to ping the Webfaction Django forum > > > (http://forum.webfaction.com/viewforum.php?id=19) as they're pretty > > > good about responding, and one of the admins might check if something > > > is wrong with your config. > > > > And I *believe* the "3 small sites" comes from the fact that on > > > Shared1, a max of 3 Apache processes will run, so you can run a site > > > on each process. I might be completely wrong on that though. > > > > Jay P --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
What is important to understand with Apache is that there is a parent process, which effectively acts as a supervisor process, and the child processes which actually accept and handle requests. When one uses 'restart' with the traditional 'apachectl' management command, it only kills off and restarts the child processes and does not stop the parent process. Because the child processes are a fork of the parent process, if for some reason there was a memory leak in the parent process, that leak would be inherited by the child process on a 'restart'. To fully kill off the parent and the children, you would do a 'stop' followed by a 'start' if using 'apachectl'. As to what the Webfaction script does I can only speculate. What you need to thus do is get 'ps' to also output the 'PPID' value using 'l' flag so you can work out which is the parent 'httpd' process. For example: $ ps axlwww | egrep '(PPID|httpd)' UID PID PPID CPU PRI NI VSZRSS WCHAN STAT TT TIME COMMAND 0 429 1 0 31 031364 2004 - Ss?? 0:00.11 /usr/local/apache-2.2.4/bin/httpd -k start 70 430 429 0 31 030524592 - S ?? 0:00.00 /usr/local/apache-2.2.4/bin/httpd -k start 501 431 429 0 31 031136852 - S ?? 0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start 501 432 429 0 31 031136848 - S ?? 0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start 70 433 429 0 31 045296 1172 - S ?? 0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start 70 434 429 0 31 045296 1176 - S ?? 0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start The parent process here is 429. Normally this process should only use quite small amounts of private memory. I have never seen an instance of where the parent process ballooned out in size, but if you have a dodgy Apache module it feasibly might. For example, from 'top': PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE 434 httpd0.0% 0:00.01 275593 496K 2.73M 1.15M 44.2M 433 httpd0.0% 0:00.01 275593 496K 2.73M 1.14M 44.2M 432 httpd0.0% 0:00.00 11037 204K 2.50M 848K 30.4M 431 httpd0.0% 0:00.00 11037 208K 2.50M 852K 30.4M 430 httpd0.0% 0:00.00 11033 188K 2.02M 592K 29.8M 429 httpd0.0% 0:00.11 1113860K 2.73M 1.96M 30.6M Ie., 60Kbytes here only, versus 200-500Kbytes for children. Anyway, work out whether how you are restarting Apache is killing of the parent process. If not work out how to restart Apache so it is. Also look at how much memory the parent is using and see if it is small. This at least is some sanity checks you can do. BTW, what Apache modules are actually being loaded using LoadModule directive. Graham On May 30, 11:42 am, chrominance <[EMAIL PROTECTED]> wrote: > I've restarted apache several times, using the restart command in the > webapps/django directory provided by Webfaction. Perhaps there's a > more absolute restart command I don't know about, but I'm pretty sure > that's the one--whenever I use it, memory usage seems to go down. > > Also, I'm not sure if this is relevant, but whenever I do restart the > apache processes, memory usage doesn't reset to the same baseline > level--for example, the site ran just fine with about 30-40MB of RAM > over the weekend, but restarting apache now only gets us down to about > 65MB. And upon my last check the apache processes peaked at about > 85MB. If there's a memory leak in my application, it wouldn't still > leak after I've restarted apache, would it? > > I'll try the Webfaction forum too. Thanks, and keep the advice coming! > > On May 29, 8:15 pm, "Jay Parlar" <[EMAIL PROTECTED]> wrote: > > > On 5/29/07, chrominance <[EMAIL PROTECTED]> wrote: > > > > I've recently put up a newspaper site on Webfaction that was developed > > > without much concern for memory limits--coming from PHP, my knowledge > > > of memory issues is practically nil. Of course, Webfaction's plans all > > > have memory limits, and we're currently on Shared 1, which imposes a > > > 40MB limit. We're running the standard apache2/mod_pythonconfig, with > > > media served via a separate apache2 process as Webfaction and the > > > Django site recommends. The Debug setting has been set to False. I > > > believe those are the standard steps recommended for reducing resource > > > usage. > > > Did you remember to restart Apache after setting Debug to False? I > > have a site running on Share
Re: Django, shared hosting memory limits and memory optimization
I've restarted apache several times, using the restart command in the webapps/django directory provided by Webfaction. Perhaps there's a more absolute restart command I don't know about, but I'm pretty sure that's the one--whenever I use it, memory usage seems to go down. Also, I'm not sure if this is relevant, but whenever I do restart the apache processes, memory usage doesn't reset to the same baseline level--for example, the site ran just fine with about 30-40MB of RAM over the weekend, but restarting apache now only gets us down to about 65MB. And upon my last check the apache processes peaked at about 85MB. If there's a memory leak in my application, it wouldn't still leak after I've restarted apache, would it? I'll try the Webfaction forum too. Thanks, and keep the advice coming! On May 29, 8:15 pm, "Jay Parlar" <[EMAIL PROTECTED]> wrote: > On 5/29/07, chrominance <[EMAIL PROTECTED]> wrote: > > > > > I've recently put up a newspaper site on Webfaction that was developed > > without much concern for memory limits--coming from PHP, my knowledge > > of memory issues is practically nil. Of course, Webfaction's plans all > > have memory limits, and we're currently on Shared 1, which imposes a > > 40MB limit. We're running the standard apache2/mod_python config, with > > media served via a separate apache2 process as Webfaction and the > > Django site recommends. The Debug setting has been set to False. I > > believe those are the standard steps recommended for reducing resource > > usage. > > Did you remember to restart Apache after setting Debug to False? I > have a site running on Shared1, and the memory usage was close to what > you have, until I set Debug to False and restarted. > > Also, you might want to ping the Webfaction Django forum > (http://forum.webfaction.com/viewforum.php?id=19) as they're pretty > good about responding, and one of the admins might check if something > is wrong with your config. > > And I *believe* the "3 small sites" comes from the fact that on > Shared1, a max of 3 Apache processes will run, so you can run a site > on each process. I might be completely wrong on that though. > > Jay P --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django, shared hosting memory limits and memory optimization
On 5/29/07, chrominance <[EMAIL PROTECTED]> wrote: > > I've recently put up a newspaper site on Webfaction that was developed > without much concern for memory limits--coming from PHP, my knowledge > of memory issues is practically nil. Of course, Webfaction's plans all > have memory limits, and we're currently on Shared 1, which imposes a > 40MB limit. We're running the standard apache2/mod_python config, with > media served via a separate apache2 process as Webfaction and the > Django site recommends. The Debug setting has been set to False. I > believe those are the standard steps recommended for reducing resource > usage. Did you remember to restart Apache after setting Debug to False? I have a site running on Shared1, and the memory usage was close to what you have, until I set Debug to False and restarted. Also, you might want to ping the Webfaction Django forum (http://forum.webfaction.com/viewforum.php?id=19) as they're pretty good about responding, and one of the admins might check if something is wrong with your config. And I *believe* the "3 small sites" comes from the fact that on Shared1, a max of 3 Apache processes will run, so you can run a site on each process. I might be completely wrong on that though. Jay P --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Django, shared hosting memory limits and memory optimization
I've recently put up a newspaper site on Webfaction that was developed
without much concern for memory limits--coming from PHP, my knowledge
of memory issues is practically nil. Of course, Webfaction's plans all
have memory limits, and we're currently on Shared 1, which imposes a
40MB limit. We're running the standard apache2/mod_python config, with
media served via a separate apache2 process as Webfaction and the
Django site recommends. The Debug setting has been set to False. I
believe those are the standard steps recommended for reducing resource
usage.
Here's a look at the ps output:
PID RSS COMMAND
17122 19652 /home/---/webapps/django/apache2/bin/httpd -f /home/---/
webapps/django/apache2/conf/httpd.conf
19823 32232 /home/---/webapps/django/apache2/bin/httpd -f /home/---/
webapps/django/apache2/conf/httpd.conf
19824 30612 /home/---/webapps/django/apache2/bin/httpd -f /home/---/
webapps/django/apache2/conf/httpd.conf
4253 1808 sshd: [EMAIL PROTECTED]/2
4254 1456 -bash
5204 936 /bin/sh /home/---/scripts/mem-usage
5205 764 ps -u --- -o pid,rss,command
5206 708 awk {print $0}{sum+=$2} END {print "Total", sum}
Total 88168
Presumably the three apache2 processes are the Django processes that
actually count against the memory usage limit; they add up to about
80MB. Clearly there is a problem here. Because we don't yet have the
domain for the site, traffic has been extremely low (basically just
one person entering data into the site); I fear for the day we open up
the site to everyone, mainly because I have no clue how memory usage
will rocket when people start visiting the site on a regular basis.
I also have no conception of what is "normal" memory usage for an
application. For all I know, the complexity of the site may simply
demand 80MB of RAM, and there might be nothing I can do about it. How
I go about figuring that out, though, is a big mystery for me.
So I guess my questions are: how do I optimize my Django setup to use
less memory? Should I be looking at the Django project I built for
ways to optimize the code, and what should I look for? Is there a
general guide on optimizing Django applications, or at least some tips
and tricks? And how can I audit the project I've built to figure out
if 80MB is actually reasonable or if I should be able to crush it down
further?
(P.S. If someone has any idea what Webfaction means when it says you
should be able to run "3 small Django sites" in 40MB, I'd love to
know. What constitutes a "small site," anyways?)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
