Re: django ORM memory leaks in standalone script
hello kanniga,
I use correct user and password.
I can get data from database.
I read about it in an one post that described what may be due not
correct settings of connect to database, but there were no details.
On Jun 5, 2:54 pm, kanniga sivasubramanian
wrote:
> hello yakovenko sir,
> I think your databse_user and database_password are incorrect.
> I had affected by this problem. You will give correct username and
> password of mysql.
>
>
>
> On Sat, Jun 5, 2010 at 1:02 AM, tmitchell wrote:
> > Holy reading comprehension, Batman. Just saw you have DEBUG off,
> > ignore previous comment.
>
> > On Jun 4, 10:34 am, Кирилл Яковенко
> > wrote:
> >> I have tried this method, but it didn't give any results.
> >> I suspect that this is due to incorrect settings of the database or connect
> >> to it.
>
> >> 2010/6/4 Dj Gilcrease
>
> >> > On Fri, Jun 4, 2010 at 6:44 AM, yakovenko wrote:
> >> > > import os
> >> > > import sys
> >> > > os.environ['DJANGO_SETTINGS_MODULE'] = 'md.settings'
> >> > > sys.path.append('/md/lib')
> >> > > from django.db import close_connection, reset_queries
> >> > > from md.mddata.models import Info
> >> > > def test():
> >> > > for i in Info.objects.all():
> >> > > pass
>
> >> > do
>
> >> > for i in Info.objects.all().iterator(): #or just .iterator() forget which
>
> >> > and you wont leak
>
> >> > --
> >> > You received this message because you are subscribed to the Google Groups
> >> > "Django users" group.
> >> > To post to this group, send email to django-us...@googlegroups.com.
> >> > To unsubscribe from this group, send email to
> >> > django-users+unsubscr...@googlegroups.com >> > groups.com>
> >> > .
> >> > For more options, visit this group at
> >> >http://groups.google.com/group/django-users?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django users" group.
> > To post to this group, send email to django-us...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > django-users+unsubscr...@googlegroups.com.
> > For more options, visit this group
> > athttp://groups.google.com/group/django-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
Re: django ORM memory leaks in standalone script
hello yakovenko sir,
I think your databse_user and database_password are incorrect.
I had affected by this problem. You will give correct username and
password of mysql.
On Sat, Jun 5, 2010 at 1:02 AM, tmitchell wrote:
> Holy reading comprehension, Batman. Just saw you have DEBUG off,
> ignore previous comment.
>
> On Jun 4, 10:34 am, Кирилл Яковенко
> wrote:
>> I have tried this method, but it didn't give any results.
>> I suspect that this is due to incorrect settings of the database or connect
>> to it.
>>
>> 2010/6/4 Dj Gilcrease
>>
>>
>>
>> > On Fri, Jun 4, 2010 at 6:44 AM, yakovenko wrote:
>> > > import os
>> > > import sys
>> > > os.environ['DJANGO_SETTINGS_MODULE'] = 'md.settings'
>> > > sys.path.append('/md/lib')
>> > > from django.db import close_connection, reset_queries
>> > > from md.mddata.models import Info
>> > > def test():
>> > > for i in Info.objects.all():
>> > > pass
>>
>> > do
>>
>> > for i in Info.objects.all().iterator(): #or just .iterator() forget which
>>
>> > and you wont leak
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Django users" group.
>> > To post to this group, send email to django-us...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > django-users+unsubscr...@googlegroups.com> > groups.com>
>> > .
>> > For more options, visit this group at
>> >http://groups.google.com/group/django-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-us...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
Re: django ORM memory leaks in standalone script
Holy reading comprehension, Batman. Just saw you have DEBUG off,
ignore previous comment.
On Jun 4, 10:34 am, Кирилл Яковенко
wrote:
> I have tried this method, but it didn't give any results.
> I suspect that this is due to incorrect settings of the database or connect
> to it.
>
> 2010/6/4 Dj Gilcrease
>
>
>
> > On Fri, Jun 4, 2010 at 6:44 AM, yakovenko wrote:
> > > import os
> > > import sys
> > > os.environ['DJANGO_SETTINGS_MODULE'] = 'md.settings'
> > > sys.path.append('/md/lib')
> > > from django.db import close_connection, reset_queries
> > > from md.mddata.models import Info
> > > def test():
> > > for i in Info.objects.all():
> > > pass
>
> > do
>
> > for i in Info.objects.all().iterator(): #or just .iterator() forget which
>
> > and you wont leak
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django users" group.
> > To post to this group, send email to django-us...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > django-users+unsubscr...@googlegroups.com > groups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/django-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
Re: django ORM memory leaks in standalone script
Are you running with DEBUG on? I've noticed the same memory growth in
standalone scripts when in DEBUG mode.
It may be related to this:
http://docs.djangoproject.com/en/dev/ref/templates/api/#django-core-context-processors-debug
On Jun 4, 10:34 am, Кирилл Яковенко
wrote:
> I have tried this method, but it didn't give any results.
> I suspect that this is due to incorrect settings of the database or connect
> to it.
>
> 2010/6/4 Dj Gilcrease
>
>
>
> > On Fri, Jun 4, 2010 at 6:44 AM, yakovenko wrote:
> > > import os
> > > import sys
> > > os.environ['DJANGO_SETTINGS_MODULE'] = 'md.settings'
> > > sys.path.append('/md/lib')
> > > from django.db import close_connection, reset_queries
> > > from md.mddata.models import Info
> > > def test():
> > > for i in Info.objects.all():
> > > pass
>
> > do
>
> > for i in Info.objects.all().iterator(): #or just .iterator() forget which
>
> > and you wont leak
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django users" group.
> > To post to this group, send email to django-us...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > django-users+unsubscr...@googlegroups.com > groups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/django-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
Re: django ORM memory leaks in standalone script
I have tried this method, but it didn't give any results.
I suspect that this is due to incorrect settings of the database or connect
to it.
2010/6/4 Dj Gilcrease
> On Fri, Jun 4, 2010 at 6:44 AM, yakovenko wrote:
> > import os
> > import sys
> > os.environ['DJANGO_SETTINGS_MODULE'] = 'md.settings'
> > sys.path.append('/md/lib')
> > from django.db import close_connection, reset_queries
> > from md.mddata.models import Info
> > def test():
> > for i in Info.objects.all():
> > pass
>
> do
>
> for i in Info.objects.all().iterator(): #or just .iterator() forget which
>
> and you wont leak
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-us...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
Re: django ORM memory leaks in standalone script
On Fri, Jun 4, 2010 at 6:44 AM, yakovenko wrote:
> import os
> import sys
> os.environ['DJANGO_SETTINGS_MODULE'] = 'md.settings'
> sys.path.append('/md/lib')
> from django.db import close_connection, reset_queries
> from md.mddata.models import Info
> def test():
> for i in Info.objects.all():
> pass
do
for i in Info.objects.all().iterator(): #or just .iterator() forget which
and you wont leak
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
django ORM memory leaks in standalone script
Hello guys,
I use django ORM in standalone daemon. And It sucks more and more
memory.
I have created small script for testing:
---
import os
import sys
os.environ['DJANGO_SETTINGS_MODULE'] = 'md.settings'
sys.path.append('/md/lib')
from django.db import close_connection, reset_queries
from md.mddata.models import Info
def test():
for i in Info.objects.all():
pass
test()
reset_queries()
close_connection()
---
This script sucks about 400 Mb and doesn't return theirs.
Additional info: I use Debian 2.6.26-2-amd64. DEBUG is False in
settings.py.
What do I do wrong?
Could this be due to not properly configure the connection?
Setting of db is
DATABASE_ENGINE = 'mysql'
DATABASE_NAME = 'md'
DATABASE_USER = 'md'
DATABASE_PASSWORD = 'md'
DATABASE_HOST = ''
DATABASE_PORT = ''
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
Re: Memory Leaks
On Sat, 2009-02-07 at 19:27 -0800, python6009 wrote: > Hi, > > I am using matplotlib/pyplot on my site to dynamically generate PNG > plots. And I am experiencing dramatic memory leaks. Within 10-15 > hits, my Apache process grows from 15-20M to 100M. Let's be accurate with the terminology here, please. Nothing in your description suggests you have a memory leak. It suggests that you have memory *usage*, but that's not the same thing. A leak is when something reserves some memory and never returns it to the general pool. And you haven't shown that that is happening here. If you need 100M to process the image, then it's likely that the Apache process that is doing that work will grow to 100M. The way malloc() and free() work in Unix, too, means that if 100M is reserved by Apache via malloc(), a subsequent free() call will let Apache reuse that memory, but won't return it to the system until the process terminates -- which happens regularly with Apache children, so it's only a transient usage. Make sure you approach this with the right debugging attitude: you're initially trying to discover what is *using* the memory. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Memory Leaks
On Sat, Feb 7, 2009 at 10:27 PM, python6009 wrote:
>
> Hi,
>
> I am using matplotlib/pyplot on my site to dynamically generate PNG
> plots. And I am experiencing dramatic memory leaks. Within 10-15
> hits, my Apache process grows from 15-20M to 100M.
>
> I am using Django 1.0.2-final, Apache 2.2.1, Python 2.4.3, matplotlib
> 0.98.5.2. The leak happens under both Apache (with mod_wsgi 2.3) and
> the development server. My OS is RHEL5.
>
> Below is a simple code snippet that causes the leak. Please let me
> know if I am doing something wrong or if there is a better way to
> write this. Thanks.
>
>
>
> from matplotlib import pyplot
>
> def test_graph (request):
>
>f = pyplot.figure()
>ax = f.add_subplot(111)
>ax.plot([1,2,3])
>
>ax.fill_between([1,2,3],[1,2,3],[1.1,2.1,3.1])
>
>ax.grid(True)
>ax.legend(['hello'],
> 'upper right', shadow=True, fancybox=True)
>ax.set_xlabel('Time')
>ax.set_ylabel('Value ')
>
>f.text(.5, 0.93, 'my title', horizontalalignment='center')
>
>response = HttpResponse(content_type='image/png')
>
> ### both ways causes a leak
>
>f.savefig( response, format = 'png' )
> OR
>canvas = FigureCanvas(f)
>canvas.print_png(response)
>canvas = None
>
>ax = None
>f = None
>
>return response
>
>
> >
>
Django isn't known to leak memory and nothing you're doing here is
particularly uncommon, or outlandish so I'd guess the issue is with
matplotlib, I'd try asking on their mailing list if there are any known
issues.
Alex
--
"I disapprove of what you say, but I will defend to the death your right to
say it." --Voltaire
"The people's good is the highest law."--Cicero
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
Memory Leaks
Hi,
I am using matplotlib/pyplot on my site to dynamically generate PNG
plots. And I am experiencing dramatic memory leaks. Within 10-15
hits, my Apache process grows from 15-20M to 100M.
I am using Django 1.0.2-final, Apache 2.2.1, Python 2.4.3, matplotlib
0.98.5.2. The leak happens under both Apache (with mod_wsgi 2.3) and
the development server. My OS is RHEL5.
Below is a simple code snippet that causes the leak. Please let me
know if I am doing something wrong or if there is a better way to
write this. Thanks.
from matplotlib import pyplot
def test_graph (request):
f = pyplot.figure()
ax = f.add_subplot(111)
ax.plot([1,2,3])
ax.fill_between([1,2,3],[1,2,3],[1.1,2.1,3.1])
ax.grid(True)
ax.legend(['hello'],
'upper right', shadow=True, fancybox=True)
ax.set_xlabel('Time')
ax.set_ylabel('Value ')
f.text(.5, 0.93, 'my title', horizontalalignment='center')
response = HttpResponse(content_type='image/png')
### both ways causes a leak
f.savefig( response, format = 'png' )
OR
canvas = FigureCanvas(f)
canvas.print_png(response)
canvas = None
ax = None
f = None
return response
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
Re: Memory leaks in DB API
Hi, Dody, Yes, I did see your reply. However, since I didn't see a public reply saying "Yes, that's it. Thank you!". I assumed that your reply wasn't the final solution and the thread was still open. Did Vidar reply to you saying this was indeed the solution? This subject line and the mention about running out of memory in the original post caused a knee-jerk "Yikes!" reaction for me, based on my all-too-real experiences with memory leaks. I'll try to take a deep breath and relax the next time I see "memory leak" in a Django thread. Unless of course, it's about a very real memory leak. Django-ly Yours, - Jason
Re: Memory leaks in DB API
Didn't you see my reply? His problem wasn't memory leak at all. It was just a case of django (on debug mode) filling up list with debugging info, which could fill up when many sql queries being done in a single request/response cycle. Jason Huggins wrote: I don't know what is wrong in this particular scenario.. but all I have to say is.... Memory leaks should be taken very seriously and should be hunted down and exterminated with extreme prejudice.
Re: Memory leaks in DB API
Good to know. Then I stand corrected. But I guess this proves the point that I *still* don't know what the heck is causing my very real and very "now" memory leak problems in my Plone app :-) (And I should probably switch my rant to the Plone boards, not here). I like to think to that I'm relatively smart guy, but tracking down memory leaks in Python truly humbles me. -Jason
Re: Memory leaks in DB API
On 2/7/06, Jason Huggins <[EMAIL PROTECTED]> wrote: Yes, Python has built-in garbage collection, but is way to easy tocreate circular references between objects (parent references child,child references parent)... Once that circular reference is createdbetween two objects, the objects will never be garbage collected by Python, and thus, a memory leak is born. Python 2.0 added cycle detection about 5 years ago.-Jonathan
Re: Memory leaks in DB API
I don't know what is wrong in this particular scenario.. but all I have to say is.... Memory leaks should be taken very seriously and should be hunted down and exterminated with extreme prejudice. Yes, Python has built-in garbage collection, but is way to easy to create circular references between objects (parent references child, child references parent)... Once that circular reference is created between two objects, the objects will never be garbage collected by Python, and thus, a memory leak is born. If the code that spawns those circular references gets called over and over again (very common if its part of SQL read/writes), the problem won't get "fixed" until the app uses up all available RAM in the OS, crashes, and a system admin (or script written by the admin) "reboots" the app to start the cycle all over again. There is nothing worse than a slow memory leak that uses up all your system memory over the course of days, weaks, (even months depending on usage) and eventually crashes your site and/or your OS. (This is happening to me now with a Zope/Plone app of mine.) The problem is so hard to find, debug, and fix, that my current solution is to simply reboot my Plone app every week to "fix" the memory leak problem. Very crufty and I hate it. The best solution for fixing memory leaks is prevention and early detection. ... and treating them like the evil cancer cells that they are. On that note, good luck, and have a nice day! :-) - Jason
Re: Memory leaks in DB API
Your setting might be using DEBUG = True, which save all SQL queries and
its start/stop time for debugging when something goes wrong. Try turning
it to False.
[EMAIL PROTECTED] wrote:
Hi all
I am wondering if other people have noticed an apparent bug in Django
when using the DB API to insert a large number (~1 million) of records
(each record ~2k) into MySQL (WinXP/Python 2.3).
Here is a simplified/clean version of the code I am running:
from django.models.appname import foos
filelist= glob.glob("path/*.dat")
for file in filelist:
f = open(file)
data = f.read()
f.close()
records = parseData(data)
for r in records:
myfoo = foos.Foo(myfield=r)
myfoo.save()
When I run this on this large chunk of data the memory footprint of the
Python process continues to climb until the machine runs out of memory.
If the last two lines however are replaced by a direct MySQLdb
connection the memory is stable at approximately 4mb and finishes
without problems.
I have tried to set the "f" and "foo" variables to None but it doesn't
help.
Any ideas what is wrong ?
br
Vidar Masson
Memory leaks in DB API
Hi all
I am wondering if other people have noticed an apparent bug in Django
when using the DB API to insert a large number (~1 million) of records
(each record ~2k) into MySQL (WinXP/Python 2.3).
Here is a simplified/clean version of the code I am running:
from django.models.appname import foos
filelist= glob.glob("path/*.dat")
for file in filelist:
f = open(file)
data = f.read()
f.close()
records = parseData(data)
for r in records:
myfoo = foos.Foo(myfield=r)
myfoo.save()
When I run this on this large chunk of data the memory footprint of the
Python process continues to climb until the machine runs out of memory.
If the last two lines however are replaced by a direct MySQLdb
connection the memory is stable at approximately 4mb and finishes
without problems.
I have tried to set the "f" and "foo" variables to None but it doesn't
help.
Any ideas what is wrong ?
br
Vidar Masson
