[issue1208304] urllib2's urlopen() method causes a memory leak
BULOT added the comment:
Hello,
I'm facing a urllib2 memory leak issue in one of my scripts that is not
threaded. I made a few tests in order to check what was going on and I
found this already existing bug thread (but old).
I'm not able to figure out what is the issue yet, but here are a few
informations:
Platform: Debian
Python version 2.5.4
I made a script (2 attached files) in order to make access to a web page
(http://www.google.com) every second, that monitors number of file
descriptors and memory footprint.
I also introduced the gc module (Garbage Collector) in order to retrieve
numbers of objects that are not freed (like already proposed in this
thread but more focussed on gc.DEBUG_LEAK flag)
Here are my results:
First acces output:
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
unreachable objects: 11
File descriptors number: 32
Memory: 4612
Thenth access:
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
unreachable objects: 110
File descriptors number: 32
Memory: 4680
After hundred access:
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
unreachable objects: 1100
File descriptors number: 32
Memory: 5284
Each call to urllib2.urlopen() gives 11 new unreachable objects,
increases memory footprint without giving new open files.
Do you have any idea?
With the hack proposed in message
http://bugs.python.org/issue1208304#msg60751, number of unreachable
objects goes down to 8 unreachable objects remaining, but still memory
increases.
Regards.
stephbul
PS
My urlib2leak.py test calls monitor script (not able to attach it):
#! /bin/sh
PROCS='urllib2leak.py'
RUNPID=`ps aux | grep "$PROCS" | grep -v "grep" | awk '{printf $2}'`
FDESC=`lsof -p $RUNPID | wc -l`
MEM=`ps aux | grep "$PROCS" | grep -v "grep" | awk '{printf $6 }'`
echo "File descriptors number: "$FDESC
echo "Memory: "$MEM
--
nosy: +stephbul
versions: -Python 2.6, Python 2.7
Added file: http://bugs.python.org/file14171/urllib2leak.py
___
Python tracker
<http://bugs.python.org/issue1208304>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294] Management of KeyboardInterrupt in cmd.py
BULOT added the comment: Well, I made it with a diff -ruN, it works fine on my ubuntu. It is only a ctrl-C management only, not a ctrl-D. What do you mean by broken? Regards. Stephbul 2007/10/19, Guido van Rossum <[EMAIL PROTECTED]>: > > > Guido van Rossum added the comment: > > Hmm... I don't think this is the right thing to do. The code is broken > in several ways. I recommend you find someone to help you come up with > a better patch in comp.lang.python. > > -- > keywords: +patch > > __ > Tracker <[EMAIL PROTECTED]> > <http://bugs.python.org/issue1294> > __ > Added file: http://bugs.python.org/file8578/unnamed __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1294> __Well, I made it with a diff -ruN, it works fine on my ubuntu. It is only a ctrl-C management only, not a ctrl-D.What do you mean by broken?Regards.Stephbul2007/10/19, Guido van Rossum < mailto:[EMAIL PROTECTED]">[EMAIL PROTECTED]>:Guido van Rossum added the comment: Hmm... I don't think this is the right thing to do. The code is brokenin several ways. I recommend you find someone to help you come up witha better patch in comp.lang.python.--keywords: +patch __Tracker <mailto:[EMAIL PROTECTED]">[EMAIL PROTECTED]><http://bugs.python.org/issue1294";>http://bugs.python.org/issue1294> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294] Management of KeyboardInterrupt in cmd.py
BULOT added the comment:
Hello,
Here is my patch for cmd.py
Regards
stephbul
Added file: http://bugs.python.org/file8566/cmd.py.keyboardinterrupt.patch
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue1294>
__--- backuo/cmd.py 2007-10-19 13:54:25.0 +0200
+++ cmd.py 2007-10-19 14:19:54.0 +0200
@@ -130,6 +130,13 @@
line = raw_input(self.prompt)
except EOFError:
line = 'EOF'
+except KeyboardInterrupt:
+self.stdout.write('\nare you sure you want to exit? y/n ')
+answer =''
+while (answer != 'y') & (answer != 'n'):
+answer = raw_input()
+if answer == 'y':
+exit(0)
else:
self.stdout.write(self.prompt)
self.stdout.flush()
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294] Management of KeyboardInterrupt in cmd.py
New submission from BULOT: According to me, the Ctrl-C is not managed correctly in cmd.py. Ctrl-C generates a a KeyboardInterrupt exceptions, and only EOFError is managed. I propose to manage KeyboardInterrupt on line 130: print 'are you sure you want to exit? y/n' answer ='' while (answer != 'y') & (answer != 'n'): answer = raw_input() if answer == 'y': exit(0) Here is attached my new cmd.py Hope ti will help. -- components: None files: cmd.py messages: 56524 nosy: stephbul severity: normal status: open title: Management of KeyboardInterrupt in cmd.py type: behavior versions: Python 2.5 Added file: http://bugs.python.org/file8562/cmd.py __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1294> __"""A generic class to build line-oriented command interpreters. Interpreters constructed with this class obey the following conventions: 1. End of file on input is processed as the command 'EOF'. 2. A command is parsed out of each line by collecting the prefix composed of characters in the identchars member. 3. A command `foo' is dispatched to a method 'do_foo()'; the do_ method is passed a single argument consisting of the remainder of the line. 4. Typing an empty line repeats the last command. (Actually, it calls the method `emptyline', which may be overridden in a subclass.) 5. There is a predefined `help' method. Given an argument `topic', it calls the command `help_topic'. With no arguments, it lists all topics with defined help_ functions, broken into up to three topics; documented commands, miscellaneous help topics, and undocumented commands. 6. The command '?' is a synonym for `help'. The command '!' is a synonym for `shell', if a do_shell method exists. 7. If completion is enabled, completing commands will be done automatically, and completing of commands args is done by calling complete_foo() with arguments text, line, begidx, endidx. text is string we are matching against, all returned matches must begin with it. line is the current input line (lstripped), begidx and endidx are the beginning and end indexes of the text being matched, which could be used to provide different completion depending upon which position the argument is in. The `default' method may be overridden to intercept commands for which there is no do_ method. The `completedefault' method may be overridden to intercept completions for commands that have no complete_ method. The data member `self.ruler' sets the character used to draw separator lines in the help messages. If empty, no ruler line is drawn. It defaults to "=". If the value of `self.intro' is nonempty when the cmdloop method is called, it is printed out on interpreter startup. This value may be overridden via an optional argument to the cmdloop() method. The data members `self.doc_header', `self.misc_header', and `self.undoc_header' set the headers used for the help function's listings of documented functions, miscellaneous topics, and undocumented functions respectively. These interpreters use raw_input; thus, if the readline module is loaded, they automatically support Emacs-like command history and editing features. """ import string __all__ = ["Cmd"] PROMPT = '(Cmd) ' IDENTCHARS = string.ascii_letters + string.digits + '_' class Cmd: """A simple framework for writing line-oriented command interpreters. These are often useful for test harnesses, administrative tools, and prototypes that will later be wrapped in a more sophisticated interface. A Cmd instance or subclass instance is a line-oriented interpreter framework. There is no good reason to instantiate Cmd itself; rather, it's useful as a superclass of an interpreter class you define yourself in order to inherit Cmd's methods and encapsulate action methods. """ prompt = PROMPT identchars = IDENTCHARS ruler = '=' lastcmd = '' intro = None doc_leader = "" doc_header = "Documented commands (type help ):" misc_header = "Miscellaneous help topics:" undoc_header = "Undocumented commands:" nohelp = "*** No help on %s" use_rawinput = 1 def __init__(self, completekey='tab', stdin=None, stdout=None): """Instantiate a line-oriented interpreter framework. The optional argument 'completekey&
