[issue1208304] urllib2's urlopen() method causes a memory leak

[Mark Lawrence]

Mark Lawrence added the comment:

Where did file descriptors come into it, surely this is all about memory leaks? 
 In any case it's hardly a show stopper as there are at least three references 
above to the problem line of code and three workarounds.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Antoine Pitrou]

Antoine Pitrou added the comment:

I see no file descriptor leak myself:

>>> f = urllib2.urlopen("http://www.google.com";)
>>> f.fileno()
3
>>> os.fstat(3)
posix.stat_result(st_mode=49663, st_ino=5045244, st_dev=7L, st_nlink=1, 
st_uid=1000, st_gid=1000, st_size=0, st_atime=0, st_mtime=0, st_ctime=0)
>>> del f
>>> os.fstat(3)
Traceback (most recent call last):
  File "", line 1, in 
OSError: [Errno 9] Bad file descriptor

Ditto with Python 3:

>>> f = urllib.request.urlopen("http://www.google.com";)
>>> f.fileno()
3
>>> os.fstat(3)
posix.stat_result(st_mode=49663, st_ino=5071469, st_dev=7, st_nlink=1, 
st_uid=1000, st_gid=1000, st_size=0, st_atime=0, st_mtime=0, st_ctime=0)
>>> del f
>>> os.fstat(3)
Traceback (most recent call last):
  File "", line 1, in 
OSError: [Errno 9] Bad file descriptor

Furthermore, you can use the `with` statement to ensure timely disposal of 
system resources:

>>> f = urllib.request.urlopen("http://www.google.com";)
>>> with f: f.fileno()
... 
3
>>> os.fstat(3)
Traceback (most recent call last):
  File "", line 1, in 
OSError: [Errno 9] Bad file descriptor

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Ralf Schmitt]

Ralf Schmitt added the comment:

I'd consider reference cycles a bug especially if they prevent filedescriptors 
from being closed. please read the comments.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Antoine Pitrou]

Antoine Pitrou added the comment:

The entire description of this issue is bogus. Reference cycles are not a bug, 
since Python has a cyclic garbage collector. Closing as invalid.

--
nosy: +pitrou
resolution:  -> invalid
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Ralf Schmitt]

Changes by Ralf Schmitt :


--
nosy: +schmir

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[gdub]

Changes by gdub :


--
nosy: +gdub

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Mark Lawrence]

Mark Lawrence  added the comment:

On Windows Vista I can consistently reproduce this with 2.6 and 2.7 but not 
with 3.1 or 3.2.

--
nosy: +BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Mark Lawrence]

Changes by Mark Lawrence :


--
versions: +Python 2.6, Python 2.7, Python 3.1, Python 3.2 -Python 2.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[clemens pecinovsky]

clemens pecinovsky  added the comment:

i also ran into the problem of cyclic dependencies. i know if i would
call gc.collect() the problem would be solved, but calling gc.collect()
takes a long time. 

the problem is the cyclic dependency with
r.recv=r.read

i have fixed it localy by wrapping the addinfourl into a new class (i
called it addinfourlFixCyclRef) and overloading the close method, and
within the close method set the recv to none again.

class addinfourlFixCyclRef(addinfourl):
def close(self):
if self.fp is not None and hasattr(self.fp, "_sock"):
self.fp._sock.recv = None
addinfourl.close(self)



r.recv = r.read
fp = socket._fileobject(r, close=True)

resp = addinfourlFixCyclRef(fp, r.msg, req.get_full_url())


and when i call .close() from the response it just works. Unluckily i
had to patch even more in case there is an exception called.
For the whole fix see the attachment

--
nosy: +peci
Added file: http://bugs.python.org/file14827/urllib2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[BULOT]

BULOT  added the comment:

Hello, 

I'm facing a urllib2 memory leak issue in one of my scripts that is not
threaded. I made a few tests in order to check what was going on and I
found this already existing bug thread (but old).

I'm not able to figure out what is the issue yet, but here are a few
informations:
Platform: Debian
Python version 2.5.4

I made a script (2 attached files) in order to make access to a web page
(http://www.google.com) every second, that monitors number of file
descriptors and memory footprint.
I also introduced the gc module (Garbage Collector) in order to retrieve
numbers of objects that are not freed (like already proposed in this
thread but more focussed on gc.DEBUG_LEAK flag)

Here are my results:
First acces output:
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
unreachable objects:  11
File descriptors number: 32
Memory: 4612

Thenth access:
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
unreachable objects:  110
File descriptors number: 32
Memory: 4680

After hundred access:
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
gc: collectable 
unreachable objects:  1100
File descriptors number: 32
Memory: 5284

Each call to urllib2.urlopen() gives 11 new unreachable objects,
increases memory footprint without giving new open files.

Do you have any idea?
With the hack proposed in message
http://bugs.python.org/issue1208304#msg60751, number of unreachable
objects goes down to 8 unreachable objects remaining, but still memory
increases.

Regards.

stephbul

PS
My urlib2leak.py test calls monitor script (not able to attach it):
#! /bin/sh

PROCS='urllib2leak.py'

RUNPID=`ps aux | grep "$PROCS" | grep -v "grep" | awk '{printf $2}'`
FDESC=`lsof -p $RUNPID | wc -l`
MEM=`ps aux | grep "$PROCS" | grep -v "grep" | awk '{printf $6 }'`

echo "File descriptors number: "$FDESC
echo "Memory: "$MEM

--
nosy: +stephbul
versions:  -Python 2.6, Python 2.7
Added file: http://bugs.python.org/file14171/urllib2leak.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Senthil]

Senthil <[EMAIL PROTECTED]> added the comment:

> Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:
> 
> Reopening: I reproduce the problem consistently with both 2.6 and trunk 
> versions (not with python 3.0), on Windows XP.
> 

I think this bug is ONLY with respect to Windows Systems. 
I not able to reproduce this on the current trunk on Linux Ubuntu (
8.04). I tried 100 and 1000 instances of open and close and everytime
file descriptors goes through ESTABLISHED, SYNC_SENT and closes for
TCP connections.

And yeah, certain instances showed 'can't identify protocol' randomly.
But thats a different issue.

The original bug raised for Python 2.4 was originally raised on Linux
and it seems to have been fixed.
A Windows expert should comment on this, if this is consistently
reproducable on Windows.

--
nosy: +orsenthil

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Gregory P. Smith]

Changes by Gregory P. Smith <[EMAIL PROTECTED]>:


--
components: +Library (Lib) -Extension Modules
type:  -> resource usage
versions: +Python 2.5, Python 2.6, Python 2.7 -Python 2.4

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

Reopening: I reproduce the problem consistently with both 2.6 and trunk 
versions (not with python 3.0), on Windows XP.

--
nosy: +amaury.forgeotdarc
resolution: wont fix -> 
status: closed -> open

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Jeremy Hylton]

Jeremy Hylton <[EMAIL PROTECTED]> added the comment:

Python 2.4 is now in security-fix-only mode. No new features are being
added, and bugs are not fixed anymore unless they affect the stability
and security of the interpreter, or of Python applications.
http://www.python.org/download/releases/2.4.5/

This bug doesn't rise to the level of making into a 2.4.6.

--
nosy: +jhylton
resolution:  -> wont fix
status: open -> closed

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Toshio Kuratomi]

Toshio Kuratomi <[EMAIL PROTECTED]> added the comment:

One further data point.   On two rhel5 systems with identical kernels,
both x86_64, both python-2.4.3... basically, everything I've thought to
check identical, I ran the test code with f.read() in an infinite loop.
 One system only has one TCP socket in use at a time.  The other one has
multiple TCP sockets in use, but they all close eventually.

/usr/sbin/lsof -p INTERPRETER_PID|wc -l reported 

96 67 97 63 91 62 94 78

on subsequent runs.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1208304] urllib2's urlopen() method causes a memory leak

[Toshio Kuratomi]

Toshio Kuratomi <[EMAIL PROTECTED]> added the comment:

I tried to repeat the test in http://bugs.python.org/msg60749 and found
that the descriptors will close if you read from the file before closing.

so this leads to open descriptors::

  import urllib2
  f = urllib2.urlopen('http://www.google.com')
  f.close()

while this does not::

  import urllib2
  f = urllib2.urlopen('http://www.google.com')
  f.read(1)
  f.close()

--
nosy: +a.badger

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com