[issue29700] readline memory corruption when sys.stdin fd >= FD_SETSIZE for select()
Marien Zwart added the comment: GNU readline 7.0 introduces the problematic use of pselect(). GNU readline 6.3 is unaffected (and presumably older versions as well). I didn't check other implementations. I'll see if I can get this fixed in GNU readline. The non-select code is still there, and the NEWS file says pselect is used because SIGWINCH interrupts it, while it doesn't interrupt read(). So maybe readline can fall through to the old code if a high FD is used, since it doesn't regress functionality that much (while adding poll support to readline would run into similar portability issues as we're seeing here). -- ___ Python tracker <http://bugs.python.org/issue29700> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29700] readline memory corruption when sys.stdin fd >= FD_SETSIZE for select()
Changes by Marien Zwart : -- pull_requests: +448 ___ Python tracker <http://bugs.python.org/issue29700> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29700] readline memory corruption when sys.stdin fd >= FD_SETSIZE for select()
Marien Zwart added the comment: On Python 3, crash_readline_fdset.py does not crash for me, because its input() contains a check documented as: /* We should only use (GNU) readline if Python's sys.stdin and sys.stdout are the same as C's stdin and stdout, because we need to pass it those. */ and calls sys.stdin.getline() instead. I don't understand why this was added (eba769657a32cb08d96f021f40c79a54ade0bffc's commit message "Make input9) behave properly with the new I/O library" does not explain it). PyOS_Readline does still take sys_stdin and sys_stdout arguments, but the only callers in CPython itself pass the actual stdin and stdout. Not sure if it's still worth fixing (maybe just turn it from a crash into an error if the fd is too high, but don't add an alternative implementation?). On Python 2, I can fix it, but then I hit the same problem in readline itself (http://git.savannah.gnu.org/cgit/readline.git/tree/input.c#n518). So I suppose the next step is reporting it there, and see if they're interested in fixing it (looks like readline isn't currently using anything more fancy than select() and pselect(), and there's a few more calls to those that would probably also need fixing...). Doesn't seem useful to fix it here first. -- ___ Python tracker <http://bugs.python.org/issue29700> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29700] readline memory corruption when sys.stdin fd >= FD_SETSIZE for select()
Changes by Marien Zwart : -- nosy: +marienz ___ Python tracker <http://bugs.python.org/issue29700> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26310] Fix typo “variariables” in socketserver.py
New submission from Marien: This patch fixes a typo in socketserver.py -- assignee: docs@python components: Documentation files: fix-typo-variariables.patch keywords: patch messages: 259869 nosy: docs@python, marienfr priority: normal severity: normal status: open title: Fix typo “variariables” in socketserver.py Added file: http://bugs.python.org/file41858/fix-typo-variariables.patch ___ Python tracker <http://bugs.python.org/issue26310> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23309] Hang on interpreter shutdown if daemon thread prints to stdout
Marien Zwart added the comment: I'm afraid I don't have real-world code I can confirm is fixed, since I reported this on behalf of someone on irc in #python. I think I'd prefer the Py_FatalError version of the patch. It's definitely possible to see writes to stdout/stderr at this time. If I read the first version right, it'll succeed the ENTER_BUFFERED without actually grabbing the lock, which seems problematic. But failing ENTER_BUFFERED is probably also more problematic, as it will probably lead to another exception that Python'll try to write to stderr, which fails the same way... If stdout/stderr are in a broken state during shutdown, Python is probably better off calling Py_FatalError rather than intermittently discarding messages. -- ___ Python tracker <http://bugs.python.org/issue23309> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23309] Hang on interpreter shutdown if daemon thread prints to stdout
New submission from Marien Zwart: A script spawning a single daemon thread calling print() in a loop (like the attached) will usually hang on shutdown in Python 3.4.2 and hg rev 8d802fb6ae32. Attaching gdb at that point shows the following: (gdb) thread apply all bt Thread 1 (Thread 0x7fd927d58700 (LWP 30274)): #0 sem_wait () at ../sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x005282fe in PyThread_acquire_lock_timed (lock=0x1c5ea30, microseconds=microseconds@entry=-1, intr_flag=intr_flag@entry=0) at Python/thread_pthread.h:352 #2 0x00528414 in PyThread_acquire_lock (lock=, waitflag=waitflag@entry=1) at Python/thread_pthread.h:556 #3 0x00567e4c in _enter_buffered_busy (self=0x7fd927bc2b48) at ./Modules/_io/bufferedio.c:327 #4 buffered_flush (self=0x7fd927bc2b48, args=) at ./Modules/_io/bufferedio.c:874 #5 0x0042822a in PyObject_Call (func=0x7fd9277b69d8, arg=, kw=) at Objects/abstract.c:2086 #6 0x004290e4 in call_function_tail (args=0x7fd927b8d048, callable=0x7fd9277b69d8) at Objects/abstract.c:2124 #7 callmethod (is_size_t=1, va=0x7fff5c6cf6b0, format=0x0, func=0x7fd9277b69d8) at Objects/abstract.c:2193 #8 _PyObject_CallMethodId_SizeT (o=, name=, format=0x0) at Objects/abstract.c:2279 #9 0x0042822a in PyObject_Call (func=0x7fd9277b6990, arg=, kw=) at Objects/abstract.c:2086 #10 0x00428cc4 in call_function_tail (args=0x7fd927b8d048, callable=0x7fd9277b6990) at Objects/abstract.c:2124 #11 callmethod (is_size_t=0, va=0x7fff5c6cf7e0, format=0x5b9924 "", func=0x7fd9277b6990) at Objects/abstract.c:2193 #12 _PyObject_CallMethodId (o=o@entry=0x7fd927b5d3a8, name=name@entry=0x862b00 , format=format@entry=0x5b9924 "") at Objects/abstract.c:2238 #13 0x0050a521 in flush_std_files () at Python/pylifecycle.c:488 #14 0x0050a5aa in Py_Finalize () at Python/pylifecycle.c:550 #15 0x0041fc92 in Py_Main (argc=-1, argv=0x1) at Modules/main.c:787 #16 0x0041be3c in main (argc=2, argv=) at ./Programs/python.c:69 The daemon thread has exited, and the main thread hangs trying to flush stdout. I haven't fully tracked down what happens here, but I think it's this: - daemon thread calls ENTER_BUFFERED on stdout - daemon thread drops the GIL before writing to stdout - main thread grabs the GIL and starts exiting - main thread sets _Py_Finalizing, signaling daemon threads to exit - main thread calls flush_std_files and drops the GIL - daemon thread grabs the GIL and immediately exits, without reaching LEAVE_BUFFERED - main thread deadlocks trying to ENTER_BUFFERED the same file If that is what happens, I don't really see how to fix it (it's an example of daemon threads not releasing their resources, which the documentation warns about). But it's obviously unfortunate if merely writing to stdout/err is such a resource. -- components: Interpreter Core files: dio.py messages: 234615 nosy: marienz priority: normal severity: normal status: open title: Hang on interpreter shutdown if daemon thread prints to stdout type: behavior versions: Python 3.4, Python 3.5 Added file: http://bugs.python.org/file37835/dio.py ___ Python tracker <http://bugs.python.org/issue23309> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13007] gdbm 1.9 has new magic that whichdb does not recognize
New submission from Marien Zwart : dbm.whichdb (python 3) and whichdb.whichdb (python 2) only check for "magic == 0x13579ace" to recognize gdbm databases. The recently released gdbm 1.9 series adds 0x13579acd (for "32bit") and 0x13579acf (for "64bit") magics (see src/gdbmconst.h in the gdbm-1.9 or gdbm-1.9.1 source tree). Python's gdbm linked to gdbm 1.9 creates databases with the new magic, which causes the whichdb tests to fail. To reproduce this just build python against the new libgdbm and run test/test_dbm.py. whichdb should probably just check for the two new magic numbers too. This will cause it to recognize the new gdbm databases even if python's gdbm extension is linked to an older version of gdbm that cannot open them, but that is probably less likely to cause problems than the current behavior, where a database created by python's gdbm is not recognized as such. (I've left the "Versions" field untouched as I do not know which versions of python this should be backported to, but it affects all of them.) -- components: Library (Lib) messages: 144277 nosy: marienz priority: normal severity: normal status: open title: gdbm 1.9 has new magic that whichdb does not recognize type: behavior ___ Python tracker <http://bugs.python.org/issue13007> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9985] difflib.SequenceMatcher has slightly buggy and undocumented caching behavior
Marien Zwart added the comment: That fixes the first problem in python 2. It should do: self.matching_blocks = [Match._make(t) for t in non_adjacent] in python 3 though, or it will return an already-exhausted map object if it is called again. This leaves the problem of other code mutating the cached list (not realizing it is cached). I think it would make sense for these functions to just return a shallow copy of their cached list, but I have not thought that through much. -- ___ Python tracker <http://bugs.python.org/issue9985> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9985] difflib.SequenceMatcher has slightly buggy and undocumented caching behavior
New submission from Marien Zwart : SequenceMatcher caches the result of get_matching_blocks and get_opcodes. There are some problems with this: What get_matching_blocks caches is a list of tuples. The first call does not return that list: it returns map(Match._make, self.matching_blocks) (converting the tuples to namedtuples). Subsequent calls just return self.matching_blocks directly. Especially in python 3 and up this is weird, since the first call returns a map object while later calls return a list. This caching behavior is not documented, so calling code may mutate the returned list. One example of calling code is difflib itself: get_grouped_opcodes mutates the result of get_opcodes (a cached list). I am not sure if the right fix is to have get_grouped_opcodes copy before it mutates or to have get_opcodes return a copy. Snippet demonstrating both bugs: matcher = difflib.SequenceMatcher(a='bc', b='dc') print(list(matcher.get_matching_blocks())) # This should print the same thing, but it does not: print(list(matcher.get_matching_blocks())) print(matcher.get_opcodes()) print(list(matcher.get_grouped_opcodes())) # This should print the same thing as the previous get_opcodes() # list, but it does not: print(matcher.get_opcodes()) -- components: Library (Lib) messages: 117612 nosy: marienz priority: normal severity: normal status: open title: difflib.SequenceMatcher has slightly buggy and undocumented caching behavior type: behavior versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2 ___ Python tracker <http://bugs.python.org/issue9985> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8652] Minor improvements to the "Handling Exceptions" part of the tutorial
Marien Zwart added the comment:
Sorry for forgetting about this for so long.
The attached patch makes the following changes:
Removes the two instances I noticed of treating exception instances as
sequences. This no longer works in python 3 and is not very useful in python 2,
so best avoided.
Adds a warning explaining ``except FooException, BarException:`` does not do
what you might think. I think this is a good warning to have (because you *can*
omit those parens in many other places) and it mentions the pre-python 2.6
syntax for "as" by side effect.
There are two things I'm not entirely happy about:
I initially wrote ``print "I/O error({0.errno}): {0.strerror}".format(e)``, but
then noticed this string formatting trick is not mentioned in the previous
chapter of the tutorial. Dict access (``"{0[foo]}".format(d)``) is, but
attribute access is not. Is this worth adding to that chapter, so it can be
used here?
Binding an exception instance to a variable (using "as") is explained near the
bottom of the section on catching exceptions, well before it is used. Perhaps
this could do with a bit of reordering? I felt it better to keep my initial
patch more minimal though.
Comments? :)
--
keywords: +patch
Added file: http://bugs.python.org/file18857/errors.patch
___
Python tracker
<http://bugs.python.org/issue8652>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8652] Minor improvements to the "Handling Exceptions" part of the tutorial
New submission from Marien Zwart : Based on questions asked on freenode's #python by people reading the tutorial I would like to suggest two improvements to http://docs.python.org/tutorial/errors.html#handling-exceptions : - Mention the older "except SomeException, e:" syntax too, not just "except SomeException as e:". I have had people ask me to help them upgrade to a newer python because they thought their version of python did not support catching exception instances (their distro python being 2.5). A big fat warning that they're reading the Python 2.6 tutorial (with links to older tutorials) may also work. - Mention "except IOError as e" before mentioning "except IOError as (errno, strerror):". The latter is an advanced and relatively unusual trick, combining regular exception catching and unpacking. Those two concepts need to be explained before the combination is. I have had people ask me how to do "except KeyError as (key, insert_something_here):" because they thought they *always* needed those parens there. (perhaps just not mentioning unpacking here at all, using "except IOError as e:" and then using e.errno would make more sense, especially since "except IOError as (errno, strerror):" is a SyntaxError in python 3 and up (because of the unpacking) and python 2.5.x and down (because of the "as")...) I can try to write a patch to the documentation if you like. -- assignee: d...@python components: Documentation messages: 105218 nosy: d...@python, marienz priority: normal severity: normal status: open title: Minor improvements to the "Handling Exceptions" part of the tutorial ___ Python tracker <http://bugs.python.org/issue8652> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7336] threading module not properly handling shutdown
Changes by Marien Zwart : -- nosy: +marienz ___ Python tracker <http://bugs.python.org/issue7336> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6944] socket.getnameinfo raises SystemError on bogus input
Marien Zwart added the comment: Attaching a less broken patch, after Taggnostr pointed out on irc that the code I removed is not actually dead. I really don't know why I thought it was. I did take the opportunity to make it more obvious what that code is actually checking for. -- Added file: http://bugs.python.org/file14930/diff.patch ___ Python tracker <http://bugs.python.org/issue6944> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6944] socket.getnameinfo raises SystemError on bogus input
New submission from Marien Zwart :
socket.getnameinfo passes a PyObject* to PyArg_ParseTuple without
checking if it's a tuple first. That means it raises SystemError on
invalid input where TypeError would make more sense:
>>> socket.getnameinfo('localhost', 0)
SystemError: new style getargs format but argument is not a tuple
An explicit check for TypeError seems like the best way to fix this.
Patch (against Python 3.1.1, but it looked like this applies easily to
other branches too) attached, which also removes a bit of dead code.
--
components: Extension Modules
files: diff.patch
keywords: patch
messages: 92858
nosy: marienz
severity: normal
status: open
title: socket.getnameinfo raises SystemError on bogus input
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2
Added file: http://bugs.python.org/file14929/diff.patch
___
Python tracker
<http://bugs.python.org/issue6944>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
