[issue30385] Segfault on OSX with 3.6.1
Gunnar Aastrand Grimnes added the comment: Adding env no_proxy='*' does indeed fix it! Thanks a lot! I guess it's pretty hard to fix. Or even to detect, so that one could log a warning before dying :( -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30385> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30385] Segfault on OSX with 3.6.1
Gunnar Aastrand Grimnes added the comment: I am unable to reduce it to a small(ish) test-case like for 3.6.1, but I can also get 2.7.1 to seg fault in my original program, and the stack trace is similar: Process: Python [4023] Path: /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python Identifier:Python Version: 2.7.11 (2.7.11) Code Type: X86-64 (Native) Parent Process:Python [4018] Responsible: Python [4023] User ID: 501 Date/Time: 2017-05-17 15:04:48.824 +0200 OS Version:Mac OS X 10.12.3 (16D32) Report Version:12 Anonymous UUID:DB16C70B-F979-86B6-864C-0AC84B0E6402 Sleep/Wake UUID: 03D0B108-E764-4834-A88D-7919CAF9B786 Time Awake Since Boot: 3000 seconds System Integrity Protection: enabled Crashed Thread:0 Dispatch queue: com.apple.os.log Exception Type:EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x00010617c832 Exception Note:EXC_CORPSE_NOTIFY Termination Signal:Segmentation fault: 11 Termination Reason:Namespace SIGNAL, Code 0xb Terminating Process: exc handler [0] VM Regions Near 0x10617c832: VM_ALLOCATE00010603c000-00010617c000 [ 1280K] rw-/rwx SM=COW --> VM_ALLOCATE0001061bc000-0001062fc000 [ 1280K] rw-/rwx SM=COW Application Specific Information: crashed on child side of fork pre-exec Thread 0 Crashed:: Dispatch queue: com.apple.os.log 0 libsystem_trace.dylib 0x7fffab4bb622 _os_log_cmp_key + 4 1 libsystem_c.dylib 0x7fffab324144 rb_tree_find_node + 58 2 libsystem_trace.dylib 0x7fffab4bbb5a __os_log_create_block_invoke + 30 3 libdispatch.dylib 0x7fffab2460b8 _dispatch_client_callout + 8 4 libdispatch.dylib 0x7fffab246cf2 _dispatch_barrier_sync_f_invoke + 83 5 libsystem_trace.dylib 0x7fffab4bb8ac os_log_create + 590 6 com.apple.SystemConfiguration 0x7fff9c8faafa __SCDynamicStoreCheckRetryAndHandleError + 167 7 com.apple.SystemConfiguration 0x7fff9c8f9e12 SCDynamicStoreCopyValue + 189 8 com.apple.SystemConfiguration 0x7fff9c8fe6fb SCDynamicStoreCopyProxiesWithOptions + 560 9 _scproxy.so 0x000104cb9914 get_proxy_settings + 24 10 org.python.python 0x0001044c2bfb PyEval_EvalFrameEx + 26904 11 org.python.python 0x0001044c672d fast_function + 264 12 org.python.python 0x0001044c2af3 PyEval_EvalFrameEx + 26640 13 org.python.python 0x0001044c672d fast_function + 264 14 org.python.python 0x0001044c2af3 PyEval_EvalFrameEx + 26640 15 org.python.python 0x0001044c672d fast_function + 264 16 org.python.python 0x0001044c2af3 PyEval_EvalFrameEx + 26640 17 org.python.python 0x0001044c672d fast_function + 264 18 org.python.python 0x0001044c2af3 PyEval_EvalFrameEx + 26640 19 org.python.python 0x0001044c672d fast_function + 264 20 org.python.python 0x0001044c2af3 PyEval_EvalFrameEx + 26640 21 org.python.python 0x0001044bc0f1 PyEval_EvalCodeEx + 1583 22 org.python.python 0x000104460fb1 function_call + 352 23 org.python.python 0x000104442eb0 PyObject_Call + 99 24 org.python.python 0x0001044c3294 PyEval_EvalFrameEx + 28593 25 org.python.python 0x0001044bc0f1 PyEval_EvalCodeEx + 1583 26 org.python.python 0x000104460fb1 function_call + 352 27 org.python.python 0x000104442eb0 PyObject_Call + 99 28 org.python.python 0x0001044c3294 PyEval_EvalFrameEx + 28593 29 org.python.python 0x0001044bc0f1 PyEval_EvalCodeEx + 1583 30 org.python.python 0x000104460fb1 function_call + 352 31 org.python.python 0x000104442eb0 PyObject_Call + 99 32 org.python.python 0x00010444dcb8 instancemethod_call + 173 33 org.python.python 0x000104442eb0 PyObject_Call + 99 34 org.python.python 0x0001044c2a27 PyEval_EvalFrameEx + 26436 35 org.python.python 0x0001044c672d fast_function + 264 36 org.python.python 0x0001044c2af3 PyEval_EvalFrameEx + 26640 37 org.python.python 0x0001044c672d fast_function + 264 38 org.python.python 0x0001044c2af3 PyEval_EvalFrameEx + 26640 39 org.python.python 0x0001044bc0f1 PyEval_EvalCod
[issue30385] Segfault on OSX with 3.6.1
New submission from Gunnar Aastrand Grimnes: This script, using requests and multiprocessing, will segfault on MacOS 16.4.0 with python 3.6.1. In 3.6.0 it runs fine. The program is reduced from a much larger program, some weird things need to be present to trigger the crash. For instance the unused tornado imports on the top. Crash log: Process: Python [1510] Path: /usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python Identifier:Python Version: 3.6.1 (3.6.1) Code Type: X86-64 (Native) Parent Process:Python [1509] Responsible: Python [1510] User ID: 501 Date/Time: 2017-05-17 14:23:47.307 +0200 OS Version:Mac OS X 10.12.3 (16D32) Report Version:12 Anonymous UUID:DB16C70B-F979-86B6-864C-0AC84B0E6402 Sleep/Wake UUID: 03D0B108-E764-4834-A88D-7919CAF9B786 Time Awake Since Boot: 630 seconds System Integrity Protection: enabled Crashed Thread:0 Dispatch queue: com.apple.os.log Exception Type:EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x00010b692832 Exception Note:EXC_CORPSE_NOTIFY Termination Signal:Segmentation fault: 11 Termination Reason:Namespace SIGNAL, Code 0xb Terminating Process: exc handler [0] VM Regions Near 0x10b692832: VM_ALLOCATE00010b652000-00010b692000 [ 256K] rw-/rwx SM=COW --> shared memory 00010b6e3000-00010b6eb000 [ 32K] r--/rw- SM=SHM Application Specific Information: crashed on child side of fork pre-exec Thread 0 Crashed:: Dispatch queue: com.apple.os.log 0 libsystem_trace.dylib 0x7fffab4bb622 _os_log_cmp_key + 4 1 libsystem_c.dylib 0x7fffab324144 rb_tree_find_node + 58 2 libsystem_trace.dylib 0x7fffab4bbb5a __os_log_create_block_invoke + 30 3 libdispatch.dylib 0x7fffab2460b8 _dispatch_client_callout + 8 4 libdispatch.dylib 0x7fffab246cf2 _dispatch_barrier_sync_f_invoke + 83 5 libsystem_trace.dylib 0x7fffab4bb8ac os_log_create + 590 6 com.apple.SystemConfiguration 0x7fff9c8faafa __SCDynamicStoreCheckRetryAndHandleError + 167 7 com.apple.SystemConfiguration 0x7fff9c8f9e12 SCDynamicStoreCopyValue + 189 8 com.apple.SystemConfiguration 0x7fff9c8fe6fb SCDynamicStoreCopyProxiesWithOptions + 560 9 _scproxy.cpython-36m-darwin.so 0x00010b536911 get_proxy_settings + 24 10 org.python.python 0x00010a464081 _PyCFunction_FastCallDict + 526 11 org.python.python 0x00010a4c87d7 call_function + 489 12 org.python.python 0x00010a4c15ca _PyEval_EvalFrameDefault + 5000 13 org.python.python 0x00010a4c992e _PyFunction_FastCall + 121 14 org.python.python 0x00010a4c87ae call_function + 448 15 org.python.python 0x00010a4c15ca _PyEval_EvalFrameDefault + 5000 16 org.python.python 0x00010a4c992e _PyFunction_FastCall + 121 17 org.python.python 0x00010a4c87ae call_function + 448 18 org.python.python 0x00010a4c15ca _PyEval_EvalFrameDefault + 5000 19 org.python.python 0x00010a4c992e _PyFunction_FastCall + 121 20 org.python.python 0x00010a4c87ae call_function + 448 21 org.python.python 0x00010a4c15ca _PyEval_EvalFrameDefault + 5000 22 org.python.python 0x00010a4c992e _PyFunction_FastCall + 121 23 org.python.python 0x00010a4c87ae call_function + 448 24 org.python.python 0x00010a4c15ca _PyEval_EvalFrameDefault + 5000 25 org.python.python 0x00010a4c992e _PyFunction_FastCall + 121 26 org.python.python 0x00010a4c87ae call_function + 448 27 org.python.python 0x00010a4c15ca _PyEval_EvalFrameDefault + 5000 28 org.python.python 0x00010a4c8fa7 _PyEval_EvalCodeWithName + 1829 29 org.python.python 0x00010a4c985e _PyFunction_FastCallDict + 444 30 org.python.python 0x00010a42c4d9 _PyObject_FastCallDict + 196 31 org.python.python 0x00010a42c5fc _PyObject_Call_Prepend + 156 32 org.python.python 0x00010a42c35a PyObject_Call + 101 33 org.python.python 0x00010a4c1825 _PyEval_EvalFrameDefault + 5603 34 org.python.python 0x00010a4c8fa7 _PyEval_EvalCodeWithName + 1829 35 org.python.python 0x00010a4c0207 PyEval_EvalCodeEx + 52 36 org.python.python 0x00010a44c493 function_call + 338
[issue20528] fileinput module will read whole file into memory when using fileinput.hook_encoded due to codecs.StreamReader.readlines
New submission from Gunnar Aastrand Grimnes:
When reading large files with fileinput, it will work as expected and only
process a line at a time when used normally, but if you add an hook_encoded
openhook it will read the whole file into memory before returning the first
line.
Verify by running this program on a large text file:
import fileinput
for l in fileinput.input(openhook=fileinput.hook_encoded('iso-8859-1')):
raw_input()
and check how much memory it uses. Remove the openhook and memory usage goes
down to nothing.
The problem is that fileinput calls readlines with a size-hint and in
codecs.StreamReader, readlines explicitly ignores this hint and reads all lines
into memory.
http://bugs.python.org/issue20501 is open for fixing up the documentation for
fileinput, but a fix would also be nice.
I see two options:
1. As suggested by r.david.murray: Give us a way of signaling to fileinput that
it should not use readlines, for instance by setting buffer=None
2. Fix the codecs module to allow StreamReader to respect the hint if given.
Although the comment there says it's no efficient way to do this, at least an
inefficient way would be better than reading a possibly infinite stream in. A
simple solution would be to repeatedly call readline. A more complicated
solution would be to read chunks from the stream, and then encode them, just
like the readline method does.
BTW - this issue is py2.7 only, I tested a file object from io.open with
encoding in 3.3 and it supports readlines just fine.
--
components: IO
messages: 210371
nosy: gromgull
priority: normal
severity: normal
status: open
title: fileinput module will read whole file into memory when using
fileinput.hook_encoded due to codecs.StreamReader.readlines
versions: Python 2.7
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20528
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20501] fileinput module will read whole file into memory when using fileinput.hook_encoded
Gunnar Aastrand Grimnes added the comment: Agreed that a doc-fix is the first step. Shall I make another issue for the fix in fileinput? I also wonder if the comment in codecs is not too negative - it is not easy to get it 100% right, but the method just above readlines is readline, which does somehow manage to read a single line. To me it seems like it would be better to makes readlines repeatedly call readline. It would also be inefficient, but still better than reading the whole (possibly infinite) stream in ... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20501 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20501] fileinput module will read whole file into memory when using fileinput.hook_encoded
New submission from Gunnar Aastrand Grimnes:
When reading large files with fileinput, it will work as expected and only
process a line at a time when used normally, but if you add an hook_encoded
openhook it will read the whole file into memory before returning the first
line.
Verify by running this program on a large text file:
import fileinput
for l in fileinput.input(openhook=fileinput.hook_encoded('iso-8859-1')):
raw_input()
and check how much memory it uses. Remove the openhook and memory usage goes
down to nothing.
--
components: IO
messages: 210130
nosy: gromgull
priority: normal
severity: normal
status: open
title: fileinput module will read whole file into memory when using
fileinput.hook_encoded
type: behavior
versions: Python 2.7
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20501
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20501] fileinput module will read whole file into memory when using fileinput.hook_encoded
Gunnar Aastrand Grimnes added the comment: The problem lies in codecs.py here: http://hg.python.org/cpython/file/ae7facd874ba/Lib/codecs.py#l581 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20501 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11649] startElementNS in xml.sax.saxutils.XMLGenerator ignored encoding
Gunnar Aastrand Grimnes gromg...@gmail.com added the comment: I came across the bug when trying to write XML documents to StringIO objects. This does not work, since the output contains a mix of unicode and str objects, and StringIO gets confused. A better alternative is to use io.BytesIO, since all output should be encoded, but this also throws an error since it gets some unicode objects. The attached file breaks for both cases (comment/uncomment StringIO to test) -- Added file: http://bugs.python.org/file21389/xmltest.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11649 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11649] startElementNS in xml.sax.saxutils.XMLGenerator ignored encoding
New submission from Gunnar Aastrand Grimnes gromg...@gmail.com:
The startElementNS method in the XMLGenerator ignores the encoding set.
it does:
self._out.write(' xmlns:%s=%s' % (prefix, uri))
whereas it should have done:
self._write(' xmlns:%s=%s' % (prefix, uri))
Issue 938076 was similar to this, but for a different method.
--
components: XML
messages: 131863
nosy: gromgull
priority: normal
severity: normal
status: open
title: startElementNS in xml.sax.saxutils.XMLGenerator ignored encoding
type: behavior
versions: Python 2.6
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11649
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
