Memory Leak and Solution for Django and PostgreSQL Application

Hi Experts, Please help me out with a solution in finding Memory leak from a Django and PostgreSQL App. which is the best way to find Memory leak and what would be the solution? How to built Memory Profiler for a Django Applications? if anyone has ever created a Django Memory Profiler, pls

Re: memory leak in django app?

Hi Antonis, Le 2017-12-06 à 16:23, Antonis Christofides a écrit : The "SSL handshake failed" doesn't seem related to the alleged memory leak. Correct. I forgot to specify which SSL protocol to use. Using ab -f TLSV1 solved that problem. Now, you say you have this problem onl

Re: memory leak in django app?

The "SSL handshake failed" doesn't seem related to the alleged memory leak. Now, you say you have this problem only in production. If you restart uwsgi, does memory usage go immediately up? Does it go after a few requests? How many requests? Do you have the same data in development

Re: memory leak in django app?

Hi Antonis, My development server appears unaffected by this problem. Plus I can get heap stats using guppy, which is pretty cool. :) Both my production and development servers have __debug__ enabled in Python 2.7.13. However when using ab -c 100 to benchmark my nginx server I get: SSL han

Re: memory leak in django app?

Does this happen only in production? What about when you run a development server? What is the memory usage of your development server? Antonis Christofides http://djangodeployment.com On 2017-12-06 15:05, Etienne Robillard wrote: > Hi Antonis, > > Thank you for your reply. I installed the htop

Re: memory leak in django app?

Hi Antonis, Thank you for your reply. I installed the htop utility and found that 2 of my 4 uWSGI processes are using 882M (42.7%) of resident memory each. Theses two processes takes about  (85%) of the available RAM memory! That can explain why I get "out of memory" errors when no more memory

Re: memory leak in django app?

Hello, the amount of memory you need depends on what Django does and how many workers (instances of Django) you run (which usually depends on how many requests you are getting and how I/O intensive your Django application is). For many applications, 512 MB is enough. Why are you worried? The only

memory leak in django app?

Hi all, I'm struggling to understand how django/python may allocate and unallocate memory when used with uWSGI. I have a Debian system running Python 2.7 and uwsgi with 2GB of RAM and 2 CPUs. Is that enough RAM memory for a uWSGI/Gevent based WSGI app running Django? I'm running uWSGI with

Re: Built-in logging module may lead to memory leak?

}, > 'django.request': { > 'handlers': ['console', 'mail_admins'], > 'level': 'ERROR', > 'propagate': False > }, > 'project_name': { > 

Built-in logging module may lead to memory leak?

, 'mail_admins'], 'level': 'ERROR', 'propagate': False }, 'project_name': { 'handlers': ['console', 'logfile'], 'level': 'DEBUG', 'propagate': True,

Memory leak in 1.6.1

G'day, Since upgrading from 1.4 to 1.6.1 my app is consuming memory rather rapidly. I have this issue on both my live Ubuntu 64bit server (python 2.7.5+, django 1.6.1) and on my development Windows 7 32bit (python 2.7.3, django 1.6.1) After some diagnosis I have created a minimal test app that r

Re: memory leak in django 1.5

Just confirming that this fixed the memory leak problem for us. Thanks again :) On Friday, March 22, 2013 3:07:02 PM UTC+13, Craig de Stigter wrote: > > Karen Tracey saves the day! > > Thanks so much, seems likely that's it :) > > Craig de Stigter > > On Friday, Mar

Re: memory leak in django 1.5

raph <http://i.imgur.com/cBsUx52.png> showing memory usage >> before and after we upgraded to django 1.5 on our web server. (the upgrade >> occurs at 12:30 on the graph). >> >> I'm hoping someone else has noticed a similar thing and is able to >> provide some in

Re: memory leak in django 1.5

raph <http://i.imgur.com/cBsUx52.png> showing memory usage >> before and after we upgraded to django 1.5 on our web server. (the upgrade >> occurs at 12:30 on the graph). >> >> I'm hoping someone else has noticed a similar thing and is able to >> provide some in

Re: memory leak in django 1.5

d a similar thing and is able to provide > some insight. > https://code.djangoproject.com/ticket/19895#comment:6 notes a memory leak due to a fix that went into 1.5. The fix was reverted on the 1.5.x branch about two days ago, so one thing to try would be to run current 1.5.x branch level ra

memory leak in django 1.5

Hi everyone (cross-posted because this seems relevant to both django-users and developers, and both might have experienced this problem) We've noticed a gradual increase in memory usage for our apache processes since upgrading to django 1.5. Here is a graph showi

Re: memory leak? Am I taking crazy pills?

Ah ok. So it sounds like if I drop "print(gc.garbage)" into my code at the right points, I will be able to see where the leak is occurring? I did that and then load tested it (not via anything tricky, I'm just using "ab") via the dev server running locally, but I'm not seeing anything, it's alwa

Re: memory leak? Am I taking crazy pills?

On 22 marras, 08:27, Dan Ancona wrote: > python 2.7.3, django 1.4.2 > > Our app (a fairly simple tastypie API that runs some machine learning foo on > mongodb records) is deployed to heroku and is leaking memory at a pretty good > clip. Hitting heroku memory errors after about an hour or two,

Re: memory leak? Am I taking crazy pills?

Sorry, forgot to mention. Debug is definitely set to False! On Nov 22, 2012, at 7:44 AM, Tom Evans wrote: > On Thu, Nov 22, 2012 at 6:27 AM, Dan Ancona wrote: >> python 2.7.3, django 1.4.2 >> >> Our app (a fairly simple tastypie API that runs some machine learning foo on >> mongodb records) is

Re: memory leak? Am I taking crazy pills?

On Thu, Nov 22, 2012 at 6:27 AM, Dan Ancona wrote: > python 2.7.3, django 1.4.2 > > Our app (a fairly simple tastypie API that runs some machine learning foo on > mongodb records) is deployed to heroku and is leaking memory at a pretty good > clip. Hitting heroku memory errors after about an hou

memory leak? Am I taking crazy pills?

python 2.7.3, django 1.4.2 Our app (a fairly simple tastypie API that runs some machine learning foo on mongodb records) is deployed to heroku and is leaking memory at a pretty good clip. Hitting heroku memory errors after about an hour or two, under significant load. Wandered through some var

Re: Massive memory leak -- seeking help

On Fri, Nov 25, 2011 at 11:01 AM, Alvin Mites wrote: > the server itself is running with > 16GB of memory and over time it is all being consumed the the apache > process are you running mod_wsgi in 'embedded' mode? because in 'daemon' mode (the recommended mode in most cases, AFAIK) the python p

Massive memory leak -- seeking help

I have a Satchmo deployment w/ 500k products running on - Apache 2.2-mpm-prefork - mysql 5.1 - mod_wsgi 3.3 - mysqldb 1.2.3 Technology stacks asside (going to migrate to nginx / gunicorn / etc... over time) this site has one hell of a memory leak the server itself is running with 16GB of

Re: Possible memory leak moving to Django 1.1 from 1.0.2 in mixed use test.

After working through this I feel I need to follow through and clarify what actually happened. As you say, Russ, there does not seem to be any memory leak. However it is a lot fatter in it's memory usage. I'm guessing this is due primarily to database caching. We will be doing som

Re: Possible memory leak moving to Django 1.1 from 1.0.2 in mixed use test.

airly recent (post-beta-1) checkout of Django. > 2. What suggestions does anyone have about how I can determine what > aspect of the system is using this extra memory and holding onto it? The best advice I can point you at is Ned Batchelder's blog, when he diagnosed a memory leak he

Possible memory leak moving to Django 1.1 from 1.0.2 in mixed use test.

We've got an application that is running behind apache, mod_wsgi and front ending a postgres 8.3 db. We needed to upgrade our Satchmo to the latest version which requires Django 1.1. After the upgrade the system starts eventually running out of memory until it thrashes and locks up the machine un

Re: Subtle Memory Leak Finally Found! (DEBUG is off)

On Wed, 2008-11-19 at 11:53 +1100, Malcolm Tredinnick wrote: [...] > Once memory gets to the over-allocated stage, > it will be reclaimed... Thinking about it further, this is rubbish. Details (if you care): Memory allocated with malloc(), etc, won't necessarily be reclaimed, since it can only

Re: Subtle Memory Leak Finally Found! (DEBUG is off)

often need to request the same memory again in a short while, so it makes sense to leave it hanging around. Once memory gets to the over-allocated stage, it will be reclaimed and once the process exits, it will also be reclaimed, so there's no long-term memory leak with that model of op

Re: Subtle Memory Leak Finally Found! (DEBUG is off)

7timesTom wrote: > cars = Car.objects.filter(...) #upto 2000 items > msg =... len(cars) # can you see the massive memory use here? This is actually well documented: http://docs.djangoproject.com/en/dev/ref/models/querysets/#when-querysets-are-evaluated --~--~-~--~~~-

Re: Subtle Memory Leak Finally Found! (DEBUG is off)

27;s taken me ages to find this darn memory leak! It was > one short line that didn't even add functionality to my users. > > I have 2000 items (cars) in a database table, which my users can > search through. After the webfaction memory warning, I optimised the > code so that my

Subtle Memory Leak Finally Found! (DEBUG is off)

Some time ago Webfaction notified me "Memory usage over your limit" (80MB) It's taken me ages to find this darn memory leak! It was one short line that didn't even add functionality to my users. I have 2000 items (cars) in a database table, which my users can search through.

Re: Memory leak!

Yup - that did it. Yeeha! On Oct 16, 10:02 pm, Ned Batchelder <[EMAIL PROTECTED]> wrote: > If you have DEBUG = True, then the SQL for the queries against the > database are collected for debugging purposes.  That can consume a lot > ofmemory(especially for 800k+ records).  Could that be the prob

Re: Memory leak!

Dunno Ned - but I do, so I'll turn it off & see. THanks! On Oct 16, 10:02 pm, Ned Batchelder <[EMAIL PROTECTED]> wrote: > If you have DEBUG = True, then the SQL for the queries against the > database are collected for debugging purposes.  That can consume a lot > ofmemory(especially for 800k+

Re: Memory leak!

county_or_prefect = tokens[6]) > > zipcode.save() > > > As it's a very long data file (800,000+ records), this little > memory leak eventually gobbles all my memory. > > I know garbage collection is supposed to be automatic

Memory leak!

zipcode.save() As it's a very long data file (800,000+ records), this little memory leak eventually gobbles all my memory. I know garbage collection is supposed to be automatic...but is there anyway I can force it every once in awhile? And can anyone see some obvious no-no

Re: Another django db memory leak?

I wanted to point this out primarily to see if anyone was experiencing similar problems, or if there are any other general comments. I also am somewhat wondering if there should be bugs reported? Or am I outside the expected use cases for the db api? When I can free up time, I would be happy to

Re: Another django db memory leak?

For some reason, when I do x.save() over an over, it takes about 200k, too. either I have something really messed up, or there is a real problem with this api. On Aug 28, 10:34 am, "ristretto.rb" <[EMAIL PROTECTED]> wrote: > To summarize - I double checked that DEBUG=False and ran all the

Re: Another django db memory leak?

To summarize - I double checked that DEBUG=False and ran all the tests again: # 20k per call x = Page.objects.filter( #(Q(type='P') & (Q(status='N') | Q(status='R'))) & #(Q(last_action_datetime__isnull=True) | Q(last_action_

Another django db memory leak?

Previously I found get_or_create() to be leaky, so I stopped using it. Now I'm seeing filter() to be leaky. First, yes, I have DEBUG=False, and I have double checked that connection.queries is empty. Suppose the following model class Page(models.Model): type = models.CharField(max_length=1

Re: (DB related?) Memory leak with apache and mod_python

e else duplicate my results? Does anyone have ideas as to > what could be responsible for this memory leak? > > thanks in advance > -Mason --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users"

(DB related?) Memory leak with apache and mod_python

aked, so I think that narrows things down to mod_python (maybe the handler for it). Can anyone else duplicate my results? Does anyone have ideas as to what could be responsible for this memory leak? thanks in advance -Mason --~--~-~--~~~---~--~~ You received th

Re: Django memory leak -- or caching?

On 6/17/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > Yes, I was in debug. Both calling db.reset_queries() and setting debug > = False fixed the issue. > > Thanks everyone! I've added this to the FAQ because it comes up from time to time. http://www.djangoproject.com/documentation/faq/#why-

Re: Django memory leak -- or caching?

Yes, I was in debug. Both calling db.reset_queries() and setting debug = False fixed the issue. Thanks everyone! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send ema

Re: Django memory leak -- or caching?

On 6/17/06, Vladimir Pouzanov <[EMAIL PROTECTED]> wrote: > Looks like python's "feature" of not free'ing any memory at all. > Actually, it doesn't sound like that issue. (Info on the issue here: http://evanjones.ca/python-memory.html ) That memory release issue is specifically a problem for apps

Re: Django memory leak -- or caching?

On 6/17/06, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote: > > I've been writing some code that populates our mysql db's through > > django. I have about 55 MB of pickled data (translating to about 80k > > new table entries and 78k new users). > > > > The problem is that python keeps allocating mor

Re: Django memory leak -- or caching?

On Sat, 2006-06-17 at 04:52 +, [EMAIL PROTECTED] wrote: > Hi all, > > I've been writing some code that populates our mysql db's through > django. I have about 55 MB of pickled data (translating to about 80k > new table entries and 78k new users). > > The problem is that python keeps allocati

Re: Django memory leak -- or caching?

Hi. are you running in 'debug' or 'production' mode? debug caches alot of informatin about SQL quierues On 17/06/2006, at 2:52 PM, [EMAIL PROTECTED] wrote: > > Hi all, > > I've been writing some code that populates our mysql db's through > django. I have about 55 MB of pickled data (translating

Django memory leak -- or caching?

Hi all, I've been writing some code that populates our mysql db's through django. I have about 55 MB of pickled data (translating to about 80k new table entries and 78k new users). The problem is that python keeps allocating more and more memory --- eventually eating up over 1.1 GB at which poin

Re: Django memory leak?

On 5/25/06, Christian Schneider <[EMAIL PROTECTED]> wrote: > And > sorry for any confusion, I don't have 30 processes the output of ps was for > one process taken every 5 seconds to show the increase in memory usage. Sorry I'm an idiot. The PID was clearly the same. :) --~--~-~--~~

Re: Django memory leak?

Malcolm,thanks a lot! That really hit it.I definitely don't need DEBUG to be True because I never knew about it in the first place. Now my memory usage stays nicely constant.I already saw me re-writing the server to use hand-rolled SQL ... chris --~--~-~--~~~---~--~

Re: Django memory leak?

Jeremy,memory usage just grows until the programme dies. Thanks for the link, I'm not sure it applies as I don't know where the django objects come from. And sorry for any confusion, I don't have 30 processes the output of ps was for one process taken every 5 seconds to show the increase in memory

Re: Django memory leak?

On Wed, 2006-05-24 at 23:22 +0200, Christian Schneider wrote: > Hi all, > > I'm using django in a server that is written as a twistd application. > I've noticed that the memory footprint of the server grows heavily the > longer it runs. When I commented out all sections calling django > memory us

Re: Django memory leak?

On 5/24/06, Christian Schneider <[EMAIL PROTECTED]> wrote: > > USER PID %CPU %MEM VSZRSS TTY STAT START TIME COMMAND > dwasrod 3487 48.7 0.7 18648 15164 ? Ss 22:48 0:01 > /usr/bin/python2.3 /usr/bin/twistd -o -y jobs-application.py How big is your peak usage? Co

Django memory leak?

Hi all,I'm using django in a server that is written as a twistd application. I've noticed that the memory footprint of the server grows heavily the longer it runs. When I commented out all sections calling django memory usage stays constant. So on first sight it seems the guilt lies with Django. Be

Re: Memory leak (db connection related?) with apache/postgres and magic-removal

Seems I was wrong about the leak not being present on OSX. After further testing on OSX I have found that the behaviour is in fact the same as on Windows, and I can see the memory leak. I think I was fooled by how slow django seems to run on OSX compared with Windows. 1000 posts to a generic

Re: Memory leak (db connection related?) with apache/postgres and magic-removal

On 4/10/06, Alex Brown <[EMAIL PROTECTED]> wrote: > As a follow up to this report. I tested this program on a django > install running on Mac OSX with the same versions of everything. No > memory leaks visible. So it seems to be a Windows only issue. Although > I did notice that when I built apach

Re: Memory leak (db connection related?) with apache/postgres and magic-removal

As a follow up to this report. I tested this program on a django install running on Mac OSX with the same versions of everything. No memory leaks visible. So it seems to be a Windows only issue. Although I did notice that when I built apache for OSX it uses a different threading configuration than

Re: Memory leak (db connection related?) with apache/postgres and magic-removal

Adrian, Debug is disabled (should have mentioned this) I saw this reported and was the first thing I tried. model sent in email. Regards Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group.

Re: Memory leak (db connection related?) with apache/postgres and magic-removal

On 3/26/06, Alex Brown <[EMAIL PROTECTED]> wrote: > I have a test script that repeatedly POSTs to a generic_view > create_object form. > > The problem, is that apache's memory usage just goes up and up, and > never releases. Hi Alex, Make sure your DEBUG setting isn't set to True. If it's set to

Re: Memory leak (db connection related?) with apache/postgres and magic-removal

Alex Brown wrote: >I have a test script that repeatedly POSTs to a generic_view >create_object form. > >The problem, is that apache's memory usage just goes up and up, and >never releases. > > It's a feature :-) With DEBUG = True Django logs all executed SQL statements in db.queries. This is

Memory leak (db connection related?) with apache/postgres and magic-removal

Hi, I am running django on Windows/apache/mod_python/postgres. I am using the latest magic-removal branch with the following patches: (1) http://code.djangoproject.com/attachment/ticket/1539/magic.querydict.patch - Solves a multithreading issue with the QueryDict class (2) http://code.djangoproj