[issue17681] Work with an extra field of gzip and zip files
Change by Nikolaus Rath : -- nosy: -nikratio ___ Python tracker <https://bugs.python.org/issue17681> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can lose data from buffered files in reference cycles
Change by Nikolaus Rath : -- nosy: -nikratio ___ Python tracker <https://bugs.python.org/issue17852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34624] -W option and PYTHONWARNINGS env variable does not accept module regexes
Change by Nikolaus Rath : -- nosy: -nikratio ___ Python tracker <https://bugs.python.org/issue34624> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35561] Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s)
Nikolaus Rath added the comment: Is there a good reason to not explicitly initialize them nevertheless? Valgrind is a common tool, and this false positive will affect everyone attempting to run it on a Python extension module. -- ___ Python tracker <https://bugs.python.org/issue35561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20177] Derby #8: Convert 28 sites to Argument Clinic across 2 files
Nikolaus Rath added the comment: Sorry, no. I have long lost context and interest in this. -- ___ Python tracker <https://bugs.python.org/issue20177> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35561] Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s)
Nikolaus Rath added the comment: $ ./configure CFLAGS="-O0 -g" --with-valgrind && make -j8 still gives ==13281== Memcheck, a memory error detector ==13281== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==13281== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info ==13281== Command: /home/nikratio/clones/cpython/python /home/nikratio/in-progress/pyfuse3/test/../examples/hello.py /tmp/pytest-of-nikratio/pytest-15/test_hello_hello_py_0 ==13281== ==13281== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==13281==at 0x584906A: epoll_ctl (syscall-template.S:84) ==13281==by 0xB5C07FA: pyepoll_internal_ctl (selectmodule.c:1392) ==13281==by 0xB5C08CB: select_epoll_register_impl (selectmodule.c:1438) ==13281==by 0xB5C112A: select_epoll_register (selectmodule.c.h:599) ==13281==by 0x173031: _PyMethodDef_RawFastCallKeywords (call.c:658) ==13281==by 0x2FEFCD: _PyMethodDescr_FastCallKeywords (descrobject.c:290) ==13281==by 0x21ED25: call_function (ceval.c:4611) ==13281==by 0x21AB4E: _PyEval_EvalFrameDefault (ceval.c:3183) ==13281==by 0x21007C: PyEval_EvalFrameEx (ceval.c:533) ==13281==by 0x17245F: function_code_fastcall (call.c:285) ==13281==by 0x1728B5: _PyFunction_FastCallKeywords (call.c:410) ==13281==by 0x21EDF4: call_function (ceval.c:4634) ==13281== Address 0xffeff2d94 is on thread 1's stack ==13281== in frame #1, created by pyepoll_internal_ctl (selectmodule.c:1379) -- ___ Python tracker <https://bugs.python.org/issue35561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35561] Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s)
Nikolaus Rath added the comment: Same error with 3.7.1 and 3.6.7 (though line numbers differ slightly). -- ___ Python tracker <https://bugs.python.org/issue35561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35561] Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s)
New submission from Nikolaus Rath : With current git master, configured --with-valgrind --with-pydebug, I get: ==31074== Command: /home/nikratio/clones/cpython/python /home/nikratio/in-progress/pyfuse3/test/../examples/hello.py /tmp/pytest-of-nikratio/pytest-11/test_hello_hello_py_0 ==31074== ==31074== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==31074==at 0x584906A: epoll_ctl (syscall-template.S:84) ==31074==by 0xBDAA493: pyepoll_internal_ctl (selectmodule.c:1392) ==31074==by 0xBDAA59F: select_epoll_register_impl (selectmodule.c:1438) ==31074==by 0xBDAA5F8: select_epoll_register (selectmodule.c.h:599) ==31074==by 0x174E15: _PyMethodDef_RawFastCallKeywords (call.c:658) ==31074==by 0x300BCA: _PyMethodDescr_FastCallKeywords (descrobject.c:290) ==31074==by 0x21FC05: call_function (ceval.c:4611) ==31074==by 0x22B5E7: _PyEval_EvalFrameDefault (ceval.c:3183) ==31074==by 0x2206FF: PyEval_EvalFrameEx (ceval.c:533) ==31074==by 0x173B61: function_code_fastcall (call.c:285) ==31074==by 0x174737: _PyFunction_FastCallKeywords (call.c:410) ==31074==by 0x21FDF4: call_function (ceval.c:4634) ==31074== Address 0xffeffeb4c is on thread 1's stack ==31074== in frame #1, created by pyepoll_internal_ctl (selectmodule.c:1379) To reproduce: $ python-dev -m pip install --user pyfuse3 # for dependencies $ git clone https://github.com/libfuse/pyfuse3.git $ valgrind --trace-children=yes "--trace-children-skip=*mount*" python-dev -m pytest test/ pyfuse3 provides a C extension module, but I believe the problem is in the interpreter core as the stacktrace does not include anything from the extension. -- components: Interpreter Core messages: 332348 nosy: nikratio priority: normal severity: normal status: open title: Valgrind reports Syscall param epoll_ctl(event) points to uninitialised byte(s) type: compile error versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue35561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34624] -W option and PYTHONWARNINGS env variable does not accept module regexes
Change by Nikolaus Rath : -- nosy: +nikratio title: -W option does not accept module regexes -> -W option and PYTHONWARNINGS env variable does not accept module regexes ___ Python tracker <https://bugs.python.org/issue34624> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34920] PYTHONWARNINGS entries are escaped
Nikolaus Rath added the comment: yes, this is a duplicate. -- ___ Python tracker <https://bugs.python.org/issue34920> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34920] PYTHONWARNINGS entries are escaped
New submission from Nikolaus Rath : According to https://docs.python.org/3/library/warnings.html#describing-warning-filters: "The meaning of each of these fields [of PYTHONWARNINGS] is as described in ." The description of the "The Warnings filter" says "module is a string containing a regular expression that the module name must match." However, when parsing PYTHONWARNINGS, the warnings module explicitly escapes the module field. -- components: Library (Lib) messages: 327272 nosy: nikratio priority: normal severity: normal status: open title: PYTHONWARNINGS entries are escaped type: behavior versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue34920> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1230540] sys.excepthook doesn't work in threads
Change by Nikolaus Rath : -- nosy: -nikratio ___ Python tracker <https://bugs.python.org/issue1230540> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can lose data from buffered files at exit
Nikolaus Rath added the comment: Given the apparent difficulties getting this right, how about not attempting to implicitly flush buffers in the finalizer at all? This means scripts relying on this will break, but in contrast to the current behavior they will break consistently so it's easy to find and fix the problem. -- ___ Python tracker <https://bugs.python.org/issue17852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31122] SSLContext.wrap_socket() throws OSError with errno == 0
Nikolaus Rath added the comment: Regarding "atrocious connection": I wish I knew, but I have no control of the connection. All I can tell is that there are frequent disconnects, occasional latency spikes, my remote ip address seems to change frequently (while the apparent local one stays as-is, so presumably some NAT in between), temporary bandwidth drops, etc. -- ___ Python tracker <http://bugs.python.org/issue31122> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31122] SSLContext.wrap_socket() throws OSError with errno == 0
New submission from Nikolaus Rath: With a particularly atrocious network connection, I often get the following exception: File "/usr/lib/python3/dist-packages/dugong/__init__.py", line 503, in connect self._sock = self.ssl_context.wrap_socket(self._sock, server_hostname=server_hostname) File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket _context=self) File "/usr/lib/python3.5/ssl.py", line 760, in __init__ self.do_handshake() File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake self._sslobj.do_handshake() File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake self._sslobj.do_handshake() OSError: [Errno 0] Error I don't think an error with errno == 0 should ever be raised by Python. -- assignee: christian.heimes components: SSL messages: 299759 nosy: christian.heimes, nikratio priority: normal severity: normal status: open title: SSLContext.wrap_socket() throws OSError with errno == 0 type: behavior versions: Python 3.5 ___ Python tracker <http://bugs.python.org/issue31122> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22499] [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636
Nikolaus Rath added the comment: Would you be willing to review a patch to incorporate the handling into the SSL module? -- ___ Python tracker <http://bugs.python.org/issue22499> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22499] [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636
Nikolaus Rath added the comment: Stefan, sorry for ignoring your earlier reply. I somehow missed the question at the end. I believe that users of the Python module are *not* expected to make use of the WANT_READ, WANT_WRITE flags. Firstly because the documentation (of Python's ssl module) doesn't say anything about that, and secondly because the code that's necessary to handle these flags is a prime example for complexity that is imposed by the C API that should be hidden to Python users. That said, could you give a more specific reference to the O'Relly book (and maybe even page or chapter)? At the moment it's a little hard for me to follow the rest of your message. Essentially, if I'm trying to write to a non-blocking, Python SSL socket, I would expect that this either succeeds or raises SSL_WANT_WRITE/READ. Not having read the book, it seems to me this is the only information that's useful to a Python caller. In what situation would you need the more exact state that your C example tracks? -- ___ Python tracker <http://bugs.python.org/issue22499> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22499] [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636
Nikolaus Rath added the comment: *ping* Just letting people know that this is still happening regularly and still present in 3.5. Some reports: https://bitbucket.org/nikratio/s3ql/issues/87/ https://bitbucket.org/nikratio/s3ql/issues/109/ (last comment) -- versions: +Python 3.5 ___ Python tracker <http://bugs.python.org/issue22499> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22499] [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636
Nikolaus Rath added the comment: This just happened again to someone else, also using Python 3.4: https://bitbucket.org/nikratio/s3ql/issues/87 Is there anything the affected people can do to help debugging this? -- ___ Python tracker <http://bugs.python.org/issue22499> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18713] Clearly document the use of PYTHONIOENCODING to set surrogateescape
Nikolaus Rath added the comment: The first thing that would come to my mind when reading Nick's proposed document (without first reading this bug report) is "So why the heck is this not the default?". It would probably save a lot of people a lot of anger if there was also a brief explanation addressing this obvious first response :-). -- nosy: +nikratio ___ Python tracker <http://bugs.python.org/issue18713> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: I believe Martin's patch (v8) is ready for a core committer review. At least I can't find anything to criticize anymore :-). -- ___ Python tracker <http://bugs.python.org/issue23529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: Except for the pointless 'myfileobj' stuff in gzip.py, rev 8 of the patch looks good to me. (Btw, I'm not actually in favor of e.g. the seekable() change. The previous patch was intended as a proof-of-concept to see what would be necessary to inherit more from DecompressReader and if it's actually worth it. But having thought about it for a while more, I don't think there's a significant gain. But I'm happy to see that you were able to cherry-pick some useful pieces out of it nevertheless). -- ___ Python tracker <http://bugs.python.org/issue23529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: As discussed in Rietveld, here's an attempt to reuse more DecompressReader for GzipFile. There is still an unexplained test failure (test_read_truncated). -- Added file: http://bugs.python.org/file38674/LZMAFile-etc.v7.patch ___ Python tracker <http://bugs.python.org/issue23529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: If you want to add support for buffer_size=0 in a separate patch/issue I think that's fine. But in that case I would not add a buffer_size parameter now at all. IMO, not having it is better as having it but not supporting zero (even if it's documented that's pretty surprising). -- ___ Python tracker <http://bugs.python.org/issue23529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: On Mar 06 2015, Martin Panter wrote: > Still to do: Need to find a better home for the _DecompressReader and > _BaseStream classes. Currently it lives in “lzma”, but apparently it > is possible for any of the gzip, bz2, lzma modules to not be > importable, so it would have to live elsewhere. Yes. > Possible options are > the io module, or a brand new internal module > (e.g. Lib/_compression.py). Thoughts? I think a new internal module would be the right choice, but I don't know what needs to be done to properly add it to the build system. So for now I'd just put it in the io module and wait for a core committer to complain :-). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- title: Limit decompressed data when reading from LZMAFile, BZ2File, GzipFile -> Limit decompressed data when reading from LZMAFile and BZ2File ___ Python tracker <http://bugs.python.org/issue23529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23529] Limit decompressed data when reading from LZMAFile and BZ2File
Nikolaus Rath added the comment: On Feb 27 2015, Martin Panter wrote: > In the code review, Nikolaus raised the idea of allowing a custom > “buffer_size” parameter for the BufferedReader. I think this would > need a bit of consideration about how it should work: > > 1. Should it be a direct wrapper around > BufferedReader(buffer_size=...)? I'm not 100% sure what you mean by that, but I think the answer is Yes. > 2. Should it also support an unbuffered reader mode like > open(buffering=0), socket.makefile(buffering=0), and > subprocess.Popen(bufsize=0)? Yes, in that case the DecompressReader should simply be used without encapsulating it in BufferedReader. > 3. Should there be any consideration for buffering in write mode > (mirroring the equivalent open(), etc parameters)? I don't think so. There's not much to be gained by buffering data before passing it to the compressor, so the only point of buffering is to avoid short writes (or reads in case of decompressing). But the low-level compressor code already avoids short writes. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- ___ Python tracker <http://bugs.python.org/issue23529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23528] Limit decompressed data when reading from GzipFile
Nikolaus Rath added the comment:
Especially now that this is only going to go into 3.5, I think it makes more
sense to handle GzipFile, LZMAFile and BZ2File all in one go. Looking at the
code, otherwise there's going to be a lot of duplication.
How about introducing a base class 'CompressedFile' that defines most of the
logic that's currently in LZMAFile (including the max_size patch from issue
23529), and having {LZMA,BZ2,Gzip}File all inherit from that base?
BZ2File and LZMAFile would probably only need to define their own constructor
to instantiate the proper compressor/decompressor object.
GzipFile would need to additionally overwrite read() and write() in order to
handle the CRC and gzip header. But I think both methods could still be written
to call super().read/write().
Did I miss something?
--
___
Python tracker
<http://bugs.python.org/issue23528>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Martin, I'll try to review your GzipFile patch. But maybe it would make sense to open a separate issue for this? I think the LZMAFile patch has not yet been reviewed or committed, and we probably want a patch for BZ2File too. The review page is already pretty cluttered right now, so I think it would make sense to use this issue for the low-level compressor/decompressor API and handle the *File API separately. -- ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Updated patch attached. -- Added file: http://bugs.python.org/file38206/issue15955_bz2_rev2.diff ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Attached is a patch for the bz2 module. -- Added file: http://bugs.python.org/file38194/issue15955_bz2.diff ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: I've started to work on the bzip module. I'll attach I work-in-progress patch if I get stuck or run out of time. -- ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: On a more practical note: I believe Nadeem at one point said that the bz2 module is not exactly an example for good stdlib coding, while the lzma module's implementation is quite clean. Therefore Therefore, one idea I had for the bz2 module was to "port" the lzma module to use the bzip2 library (instead of adding a max_size parameter to the bz2 module). Just something to consider if you find time to work on this before me. -- ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Yes, I still plan to do it, but I haven't started yet. That said, I certainly won't be offended if someone else implements the feature either. Please just let me know when you start working on this (I'll do the same), so there's no duplication of efforts. -- ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23300] httplib is using a method that doesn't exist
Nikolaus Rath added the comment: Serhiy, did you add me to Cc just for information, or is there anything I should be doing (having written the patch that introduced this bug)? -- ___ Python tracker <http://bugs.python.org/issue23300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Martin Panter writes: > We still need a patch for max_length in BZ2Decompressor, and to use it > in BZ2File. I'm still quite interested to do this. The only reason I haven't done it yet is that I'm waiting for the LZMA patch to be reviewed and committed first (no need to make any mistakes twice). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: *ping*. It's been another 8 months. It would be nice if someone could review the patch. -- ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can loose data from buffered files at exit
Nikolaus Rath added the comment: This will probably be too radial, but I think it should at least be mentioned as a possible option. We could just not attempt to implicitly flush buffers in the finalizer at all. This means scripts relying on this will break, but in contrast to the current behavior they will break consistently so it's easy to find and fix the problem. I think this is still preferable to any solution that does not guarantee flushing in 100% of the cases. -- ___ Python tracker <http://bugs.python.org/issue17852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can loose data from buffered files at exit
Nikolaus Rath added the comment: Serhiy, I believe this still happens in Python 3.4, but it is harder to reproduce. I couldn't get Armin's script to produce the problem either, but I'm pretty sure that this is what causes e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=771452#60. -- ___ Python tracker <http://bugs.python.org/issue17852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can loose data from buffered files at exit
Changes by Nikolaus Rath : -- nosy: +nikratio ___ Python tracker <http://bugs.python.org/issue17852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22499] [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636
New submission from Nikolaus Rath: I received a bugreport due to a crash when calling SSLObject.send(). The traceback ends with: [...] File "/usr/local/lib/python3.4/dist-packages/dugong-3.2-py3.4.egg/dugong/__init__.py", line 584, in _co_send len_ = self._sock.send(buf) File "/usr/lib/python3.4/ssl.py", line 679, in send v = self._sslobj.write(data) ssl.SSLError: [SSL: BAD_WRITE_RETRY] bad write retry (_ssl.c:1636) At first I thought that this is an exception that my application should catch and handle. However, when trying to figure out what exactly BAD_WRITE_RETRY means I get the impression that the fault is actually in Python's _ssl.c. The only places where this error is returned by OpenSSL are ssl/s2_pkt.c:480 and ssl/s3_pkt.c:1179, and in each case the problem seems to be with the caller supplying an invalid buffer after an initial write request failed to complete due to non-blocking IO. This does not seem to be something that could be caused by whatever Python code, so I think there is a problem in _ssl.c. -- components: Library (Lib) messages: 227582 nosy: alex, christian.heimes, dstufft, giampaolo.rodola, janssen, nikratio, pitrou priority: normal severity: normal status: open title: [SSL: BAD_WRITE_RETRY] bad write retry in _ssl.c:1636 type: crash versions: Python 3.4 ___ Python tracker <http://bugs.python.org/issue22499> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: *ping* -- ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: On 06/17/2014 01:28 PM, R. David Murray wrote: > Well, but we think it's pretty clear. This wasn't the impression that I had from the thread on python-devel, but I'll accept your judgement on that. I'll be more restrained when being asked for suggestions in the future. -- ___ Python tracker <http://bugs.python.org/issue21763> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: "R. David Murray" writes: > R. David Murray added the comment: > > Nikolaus: while I agree that Raymond's comments were a bit strongly > worded, it doesn't read to me as if the thread you link to is on point > for this issue. The thread was focused on a *specific* question, that > of calling close twice. The question of what the docs mean by "a file > like object" is a different question. Specifically, you will note > that "duck typing" never came up in that thread (as far as I > remember). That's quite correct. But did not come up in the original issue report either - that's why I still don't understand why we're discussing it here at all. My line of thought was as follows: - Standard library assumes that close() is idempotent in many places - Currently this isn't documented clearly - The best place to make it more explicit seemed to be the description of IOBase - However, changing the description of only IOBase.close() could easily give the impression that close() is somehow special, and that fewer/no assumptions are made in the standard lbirary about the presence/behavior of the other methods (e.g. fileno or readable). - Therefore, to me the best course of action seemed to add a paragraph explicitly describing that the standard library may assume that *any* method/attribute of a stream object behaves as described for IOBase. I still don't see how this contradicts / interacts with the concept of duck typing. I think the documentation (with and without the patch) clearly implies that you can implement an object that does not have the full IOBase API and successfully hand it to some standard library function - it's just that in that case you can't complain if it breaks. Or is the point of contention just the title of this issue? Maybe it was poorly chosen, I guess "Clarify standard library's expectations from stream objects" would have been better. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- ___ Python tracker <http://bugs.python.org/issue21763> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: On 06/15/2014 06:26 PM, Raymond Hettinger wrote: > Before creating more tracker items, please take time to learn about how > Python's history, [...] It'd be nice if you would have at least followed the link to http://article.gmane.org/gmane.comp.python.devel/148199 before commenting. In case you still can't spare the time for that: I explicitly asked on python-dev about this *before* I opened a tracker item. Apart from that, I think your remarks are neither appropriate nor do they belong in the tracker. -- ___ Python tracker <http://bugs.python.org/issue21763> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: Maybe I'm missing some important point here, but I think that the documentation ought to tell me how I have to design a file-like object such that it fulfills all expectations of the standard library. Yes, you can get away with less than that in many situations, but that doesn't mean that the documentation should not tell me about the full set of expectations. -- ___ Python tracker <http://bugs.python.org/issue21763> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: (refreshed patch, no changes) -- Added file: http://bugs.python.org/file35648/issue20578_r6.diff ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
Nikolaus Rath added the comment: On 06/15/2014 08:29 AM, R. David Murray wrote: > I don't think that's true, though. "file like" pretty much means "has the > file attributes that I actually use". That is, it is context dependent (duck > typing). Well, but when you pass your file-like object to some function from the standard library, you don't know what file attributes will be used. So to make sure that things work as expected, you have to make sure that your file-like object behaves as prescribed by the IOBase* classes. > I'm also not sure I see the point in the change. It is inherent in the > definition of what ABCs are. True. But not everyone reading the io documentation is familiar enough with ABCs to immediately make that mental translation. -- ___ Python tracker <http://bugs.python.org/issue21763> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: As discussed on python-devel, I'm attaching a new patch that uses memoryview.cast to ensure that the pure-Python readinto() now works with any object implementing the buffer protocol. -- Added file: http://bugs.python.org/file35647/issue20578_r5.diff ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21764] Document that IOBase.__del__ calls self.close
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file35638/iobase2.diff ___ Python tracker <http://bugs.python.org/issue21764> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21764] Document that IOBase.__del__ calls self.close
Changes by Nikolaus Rath : Removed file: http://bugs.python.org/file35637/iobase2.diff ___ Python tracker <http://bugs.python.org/issue21764> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21764] Document that IOBase.__del__ calls self.close
New submission from Nikolaus Rath: CPython's io.IOBase.__del__ calls self.close(), but this isn't documented anywhere (and may be surprised for derived classes). The attached patch extends the documentation. -- assignee: docs@python components: Documentation files: iobase2.diff keywords: patch messages: 220591 nosy: benjamin.peterson, docs@python, eric.araujo, ezio.melotti, georg.brandl, hynek, nikratio, pitrou, stutzbach priority: normal severity: normal status: open title: Document that IOBase.__del__ calls self.close type: enhancement versions: Python 3.4, Python 3.5 Added file: http://bugs.python.org/file35637/iobase2.diff ___ Python tracker <http://bugs.python.org/issue21764> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21763] Clarify requirements for file-like objects
New submission from Nikolaus Rath: It is currently not perfectly clear what Python (and the standard library) assumes about file-like objects (see e.g. http://article.gmane.org/gmane.comp.python.devel/148199). The attached doc patch tries to improve the current situation by stating explicitly that the description of IOBase et al specifies a *mandatory* interface for anything that claims to be file-like. -- assignee: docs@python components: Documentation files: iobase.diff keywords: patch messages: 220588 nosy: benjamin.peterson, docs@python, eric.araujo, ezio.melotti, georg.brandl, hynek, ncoghlan, nikratio, pitrou, stutzbach priority: normal severity: normal status: open title: Clarify requirements for file-like objects type: enhancement versions: Python 3.4, Python 3.5 Added file: http://bugs.python.org/file35636/iobase.diff ___ Python tracker <http://bugs.python.org/issue21763> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19414] iter(ordered_dict) yields keys not in dict in some circumstances
Nikolaus Rath added the comment: Raymond, it would be nice if you could respond to my last comment... -- ___ Python tracker <http://bugs.python.org/issue19414> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: Nadeem, did you get a chance to look at this? -- ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment:
I used the wrong interpreter when cutting and pasting the example above, here's
the correct version to avoid confusion with the traceback:
>>> import _pyio
>>> from array import array
>>> buf = array('b', b'x' * 10)
>>> _pyio.open('/dev/zero', 'rb').readinto(buf)
10
>>> buf = array('B', b'x' * 10)
>>> _pyio.open('/dev/zero', 'rb').readinto(buf)
Traceback (most recent call last):
File "/home/nikratio/clones/cpython/Lib/_pyio.py", line 662, in readinto
b[:n] = data
TypeError: can only assign array (not "bytes") to array slice
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/home/nikratio/clones/cpython/Lib/_pyio.py", line 667, in readinto
b[:n] = array.array('b', data)
TypeError: bad argument type for built-in operation
--
___
Python tracker
<http://bugs.python.org/issue20578>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment:
Thanks for taking the time, and apologies about the test failure. I was
probably too eager and ran only the test_io suite instead of everything.
I looked at the failure, and the problem is that the default Python
BufferedIOBase.readinto implementation is semi-broken. It should work with any
object implementing the memoryview protocol (like the C implementation), but it
really only works with bytearray objects. The testIteration test only worked
(prior to the patch) because there is a special case for array objects with
format 'b':
try:
b[:n] = data
except TypeError as err:
import array
if not isinstance(b, array.array):
raise err
b[:n] = array.array('b', data)
In other words, trying to read into any other object has always failed. In
particular, even format code 'B' fails:
>>> import _pyio
>>> from array import array
>>> buf = array('b', b'x' * 10)
>>> _pyio.open('/dev/zero', 'rb').readinto(buf)
10
>>> buf = array('B', b'x' * 10)
>>> _pyio.open('/dev/zero', 'rb').readinto(buf)
Traceback (most recent call last):
File "", line 1, in
File "/home/nikratio/clones/cpython/Lib/_pyio.py", line 1096, in readinto
buf[:len_] = array.array('b', buf2)
TypeError: bad argument type for built-in operation
The readline implementation that my patch adds for BufferedReader does not
contain this special case, and therefore with the patch even the test with a
'b'-array fails.
For now, I've added the same special casing of 'b'-type arrays to the
_readline() implementation in BufferedReader. This fixes the immediate problem
(and this time I definitely ran the entire testsuite).
However, the fix is certainly not what I would consider a good solution.. but I
guess that would better be addressed by a separate patch that also fixes the
same issue in BufferedIOBase?
--
Added file: http://bugs.python.org/file35539/issue20578_r4.diff
___
Python tracker
<http://bugs.python.org/issue20578>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17277] incorrect line numbers in backtrace after removing a trace function
Changes by Nikolaus Rath : -- nosy: +nikratio ___ Python tracker <http://bugs.python.org/issue17277> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20177] Derby #8: Convert 28 sites to Argument Clinic across 2 files
Nikolaus Rath added the comment: Tal, I was referring to this mail: https://mail.python.org/pipermail/python-dev/2014-January/132066.html -- ___ Python tracker <http://bugs.python.org/issue20177> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21555] gcmodule.c could use pytime.h
Changes by Nikolaus Rath : -- nosy: +nikratio ___ Python tracker <http://bugs.python.org/issue21555> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Nikolaus Rath added the comment: On 05/13/2014 12:41 PM, Serhiy Storchaka wrote: > Note that this is not work with the punycode encoding (and may be some > third-party encodings). I do not understand. Could you elaborate? -- ___ Python tracker <http://bugs.python.org/issue21057> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7776] http.client.HTTPConnection tunneling is broken
Nikolaus Rath added the comment: Cybjit writes: > Cybjit added the comment: > > On 2014-05-10 00:23, nikratio wrote: >> Is pip maybe doing its own certificate check, and relying on >> HTTPSConnection.host to contain the final hostname rather than the proxy? > > I think the culprit might be here > https://github.com/pypa/pip/blob/1.5.4/pip/_vendor/requests/packages/urllib3/connection.py#L172 Yes, that's the problem. I guess that nicely demonstrates why using inheritance as an API is not a good idea. I guess we nevertheless need to repair/work around this in Python 3.4? Unfortunately pip explicitly relies on _set_tunnel() to set self.host = self._tunnel_host. So we would need to change _set_tunnel() to save the original attribute somewhere, change the other methods to use the saved attribute in favor of the real one, and have connect() restore it (so that we can reconnect). This still would not allow pip to reconnect (because it overwrites the connect method), but then reconnecting a tunneled connection with pip did not work before either. Still, rather ugly. Alternatively, maybe we could also do nothing, because if pip is depending on undocumented semantics of a private method (_set_tunnel), they have to live with the consequences? Thinking about this, I think we should just revert the entire patch for 3.4, but keep it in for 3.5. That gives the pip folks enough time to fix their code. Fixing the issue in 3.4 is probably not that crucial (after all, it existed since about 2.6.3). Best, Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- ___ Python tracker <http://bugs.python.org/issue7776> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7776] http.client.HTTPConnection tunneling is broken
Nikolaus Rath added the comment: On 05/09/2014 02:02 PM, Cybjit wrote: > C:\Python34\Scripts>pip -v install simplejson > Downloading/unpacking simplejson > Could not fetch URL https://pypi.python.org/simple/simplejson/: connection > err > or: hostname 'openwrt.lan' doesn't match either of '*.c.ssl.fastly.net', > 'c.ssl. This looks as if pip tries to match the hostname in the certificate from pypi.python.org against the hostname of the local proxy. Looking at the code, I don't see why it would do that though. HTTPSConnection.connect definitely tries to match against the final hostname. Is pip maybe doing its own certificate check, and relying on HTTPSConnection.host to contain the final hostname rather than the proxy? -- ___ Python tracker <http://bugs.python.org/issue7776> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19414] iter(ordered_dict) yields keys not in dict in some circumstances
Nikolaus Rath added the comment: Raymond, I think your patch does not really address the issue reported here. The dict documentation still guarantees that mutating a dict during iteration will raise a RuntimeError or may skip elements. The OrderedDict documentation still does not point out that OrderedDicts behave differently, yet they still raise a different exception than a regular dict in the same situation. I believe it should be possible to pass an OrderedDict to any function expecting a regular dict. This is still not possible. But even if you think this is not desirable (or not worth the cost), could we at least *document* that OrderedDicts behave differently? -- ___ Python tracker <http://bugs.python.org/issue19414> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21377] PyBytes_Concat could try to concat in-place
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file35123/issue_21377_r4.diff ___ Python tracker <http://bugs.python.org/issue21377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Nikolaus Rath added the comment: Maybe. I have 1.0.1g. Could you maybe post the output of the failed test? I'd like to understand how the patch broke the test (looking at your patch alone didn't tell me much). -- ___ Python tracker <http://bugs.python.org/issue20951> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Nikolaus Rath added the comment: Antoine, are you sure this was a problem related to this patch? The test seems to work just fine for me: $ hg update -C -r b0f6983d63df $ make clean $ ./configure --with-pydebug && make -j1 $ ./python -m test -u network,urlfetch -j 8 test_poplib [1/1] test_poplib 1 test OK. Am I doing something wrong? -- ___ Python tracker <http://bugs.python.org/issue20951> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21377] PyBytes_Concat could try to concat in-place
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file35100/issue_21377_r3.diff ___ Python tracker <http://bugs.python.org/issue21377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21377] PyBytes_Concat could try to concat in-place
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file35085/issue_21377_r2.diff ___ Python tracker <http://bugs.python.org/issue21377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21377] PyBytes_Concat could try to concat in-place
Nikolaus Rath added the comment: Tentative patch attached. The test suite still passes, but I'm not sure if it actually exerts the new code path. Is there a standard way to test the C api? -- keywords: +patch Added file: http://bugs.python.org/file35084/issue_21377.diff ___ Python tracker <http://bugs.python.org/issue21377> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file35083/issue_21057_r3.diff ___ Python tracker <http://bugs.python.org/issue21057> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file35082/issue20951_r3.diff ___ Python tracker <http://bugs.python.org/issue20951> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Nikolaus Rath added the comment: As discussed on python-dev, here is a patch that changes the behavior of send() and sendall() to raise SSLWant* exceptions instead of returning zero. -- Added file: http://bugs.python.org/file35062/issue20951_r2.diff ___ Python tracker <http://bugs.python.org/issue20951> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file34864/benchmark_r3.py ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment:
Attached is an updated patch that
- removes the code duplication in _pyio.BufferedIOBase
- adds an internal _readinto helper method to _pyio.BufferedReader that makes
the implementation similar to io.BufferedReader.
- implements _pyio.BuffereadReader.{readinto,readinto1} in terms of the new
helper method and, as a side effect, also increases their performance.
Performance of the _pyio implementation on my system is:
pre-patch:
readinto: 5.130e+00 seconds
readinto1 not available
post-patch:
readinto: 2.039e+00 seconds
readinto1: 1.995e+00 seconds
--
Added file: http://bugs.python.org/file34863/issue20578_r3.diff
___
Python tracker
<http://bugs.python.org/issue20578>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10983] Errors in http.client.HTTPConnection class (python3)
Nikolaus Rath added the comment: This issue can be closed. The testcases have been added in 39ee3286d187. -- ___ Python tracker <http://bugs.python.org/issue10983> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: > Can you please extend your benchmark to also measure read and readinto? Yes - but I don't quite understand why it matters (if you need read1/readinto1, you cannot just use read/readinto instead). C readinto1: 4.638e-01 seconds C read1: 4.026e-01 seconds C readinto: 4.655e-01 seconds C read: 4.028e-01 seconds Python readinto1: 1.056e+00 seconds Python read1: 2.429e+00 seconds Python readinto: 1.895e+00 seconds Python read: 1.218e+00 seconds That shows that the Python readinto is definetely not up-to-par and could use improvement as well. Is that what you're getting at? > I'm puzzled why you are treating readinto1 differently from readinto. Maybe this is why we seem to be talking past each other :-). I did not look or work on readinto at all. All I noticed is that there is a read1, but no readinto1. So I implemented a readinto1 as well as I could. -- Added file: http://bugs.python.org/file34812/benchmark.py ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: (Rietveld is giving me errors, so I'm replying here) On 2014/04/13 02:22:23, loewis wrote: >>> Again, why a separate implementation here? >> >> For performance reasons. Relying on the default implementation >> would fall back to using read1(), which means a new bytes object >> is created first. > > Hmm. > a) if performance was relevant, it should apply to readinto() as well. I didn't even notice the readinto implementation was missing. But I agree, if we keep readinto1(), we should also add readinto(). > b) if the code is duplicated for performance only, a comment should >state that explicitly. I'm very sorry, but I still don't see which code in readinto1() is duplicate. You don't mean duplicate between io and _pyio, do you? > c) to justify a performance argument, you should really provide hard >numbers that demonstrate a performance gain justifying the code >duplication. I posted a small benchmark to the issue tracker. Personally, I think the more important argument is to keep the Python and C implementations similar. It's really nice if you can look at _pyio to find out at least roughly what happens in io. (Yes, I did put performance first in my last reply, but only because I thought you were asking about readinto1 in general, rather than the additional Python implementation in _pyio.) -- ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: Here's a little script to estimate the performance difference between using read1 and readinto1 to read large amounts of data. On my system, I get: C readinto1: 4.960e-01 seconds C read1: 4.055e-01 seconds Python readinto1: 1.066e+00 seconds Python read1: 2.565e+00 seconds In other words, _pyio.BufferedReader.readinto1 is more than a factor of 2 faster than _pyio.BufferedReader.read1 and io.readinto1 is faster than io.read1 by about 20%. On its own, I think this would justify keeping an implementation of readinto1 in _pyio.BufferedReader instead of falling back on the default (that is implemented using read1). However, I believe that people who need performance are probably not using _pyio but io, so *my* argument for keeping it implemented in _pyio is to keep the implementations similiar. I found studying _pyio very helpful to understand the C code in io. If we implement BufferedReader.readinto1 in io, but not in _pyio.BufferedReader, this advantage would be reduced. That said, I am primary interested in getting readinto1 into io. So I'm happy to either extend the patch to also provide a fast readinto implementation for _pyio (to align it with io), or to remove the readinto1 implementation in _pyio. -- Added file: http://bugs.python.org/file34811/benchmark.py ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19414] iter(ordered_dict) yields keys not in dict in some circumstances
Nikolaus Rath added the comment: The patch applies cleanly to 3.4 and 3.5, not sure why the review link does not show up. I'm attaching the file again, maybe that helps. -- Added file: http://bugs.python.org/file34804/issue19414_r2.diff ___ Python tracker <http://bugs.python.org/issue19414> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7776] http.client.HTTPConnection tunneling is broken
Nikolaus Rath added the comment: Refreshed patch. -- Added file: http://bugs.python.org/file34802/issue7776_r7_Py3.4.diff ___ Python tracker <http://bugs.python.org/issue7776> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7776] http.client.HTTPConnection tunneling is broken
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file34803/issue7776_r7_Py3.5.diff ___ Python tracker <http://bugs.python.org/issue7776> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1738] Add match parameter to filecmp.dircmp to ignore using patterns
Nikolaus Rath added the comment: Updated patch to acknowledge original authors in Misc/ACKS. -- Added file: http://bugs.python.org/file34801/issue1738_r3.diff ___ Python tracker <http://bugs.python.org/issue1738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: Thanks for the review! Attached is a new patch. I was actually pretty careful to avoid any code duplication.. are you refering to the readinto1() implementations for BytesIO and BufferedReader in Lib/_pyio.py? -- Added file: http://bugs.python.org/file34793/issue20578_r2.diff ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Nikolaus Rath added the comment: My usecase is that I have a binary stream class that internally uses memoryviews. I would like to read text data from this stream and thus encapsulate it in a TextIOWrapper. Currently, TextIOWrapper (correctly) expects read() to return bytes and fails if it receives any other bytes-like object. I can change my custom class to internally convert to bytes, but this means that the data is needlessly copied around and affects every other consumer of the class as well. Changing TextIOWrapper to work with any bytes-like object is (as far as I can see) rather simple. It does not introduce any new branches in the code, and it does not change the behavior for bytes objects at all. It does, however, eliminate unnecessary memcopies for classes that do not internally work with bytes objects. Therefore, I was hoping this patch could be considered for inclusion. The MyByteStream example that I gave in the first message is useless. I merely included it as the smallest possible code fragment that currently does not work, but would work after the patch in an attempt to illustrate what I meant - but apparently it had the opposite effect. Thanks for considering! -- type: behavior -> enhancement versions: -Python 3.4 ___ Python tracker <http://bugs.python.org/issue21057> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: Seems as if no one has an opinion on this at all: https://mail.python.org/pipermail/python-dev/2014-April/133739.html What next? -- ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file34792/issue15955_r3.diff ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15955] gzip, bz2, lzma: add option to limit output size
Nikolaus Rath added the comment: I've attached the second iteration of the patch. I've factored out a new function decompress_buf, and added two new (C only) instance variables input_buffer and input_buffer_size. I've tested the patch by enabling Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c and compiling --with-pydebug --without-pymalloc. Output is: $ valgrind --tool=memcheck --suppressions=Misc/valgrind-python.supp ./python -m test -R : -v test_lzma ==18635== Memcheck, a memory error detector ==18635== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==18635== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info ==18635== Command: ./python -m test -R : -v test_lzma ==18635== == CPython 3.5.0a0 (default:a8f3ca72f703+, Apr 11 2014, 21:48:07) [GCC 4.8.2] [] Ran 103 tests in 43.655s OK beginning 9 repetitions 123456789 [...] OK . 1 test OK. ==18635== ==18635== HEAP SUMMARY: ==18635== in use at exit: 2,328,705 bytes in 13,625 blocks ==18635== total heap usage: 2,489,623 allocs, 2,475,998 frees, 91,305,463,593 bytes allocated ==18635== ==18635== LEAK SUMMARY: ==18635==definitely lost: 0 bytes in 0 blocks ==18635==indirectly lost: 0 bytes in 0 blocks ==18635== possibly lost: 963,533 bytes in 1,334 blocks ==18635==still reachable: 1,365,172 bytes in 12,291 blocks ==18635== suppressed: 0 bytes in 0 blocks ==18635== Rerun with --leak-check=full to see details of leaked memory ==18635== ==18635== For counts of detected and suppressed errors, rerun with: -v ==18635== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2) When running the tests only once (no -R option), the number of possibly lost bytes is 544,068 in 1,131 blocks. When running without the patch, the number is 545,740 bytes in 1,134 blocks (for one iteration). I guess this means that Python is using interior pointers, so these blocks are not truly lost? -- Added file: http://bugs.python.org/file34789/issue15955_r2.diff ___ Python tracker <http://bugs.python.org/issue15955> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20375] ElementTree: Document handling processing instructions
Changes by Nikolaus Rath : -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue20375> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20375] ElementTree: Document handling processing instructions
Nikolaus Rath added the comment: Thanks for the commit! My intention is to fix the behavior itself for 3.5 (see issue 9521), so I think adding testcases for the old behavior in the meantime isn't necessary. -- ___ Python tracker <http://bugs.python.org/issue20375> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20375] ElementTree: Document handling processing instructions
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file34709/issue20375.diff ___ Python tracker <http://bugs.python.org/issue20375> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20375] ElementTree: Document handling processing instructions
Nikolaus Rath added the comment: Thanks for your feedback! I've attached an updated patch. -- Added file: http://bugs.python.org/file34708/issue20375.diff ___ Python tracker <http://bugs.python.org/issue20375> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Nikolaus Rath added the comment: Thanks for the feedback! I have attached an updated patch. I did not include any testcase because the patch did not create any new code paths, so I was assuming it'd be covered by the existing test case. But of course I was wrong. In the revised patch, I added a testcase based on your example of a more complex memoryview. (Note, however, that even with the previous implementation using nbytes = PySequence_Size(input_chunk) this test does not fail, because nbytes is used only to estimate the size of the text string). -- Added file: http://bugs.python.org/file34690/issue21057.diff ___ Python tracker <http://bugs.python.org/issue21057> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20375] ElementTree: Document handling processing instructions
Nikolaus Rath added the comment: Yes, the new testcases were deliberately included. I submitted the patch prior to the 3.4 release, am I right that at that point this wouldn't have been a problem? I have attached a new patch containing just the doc changes. I hope that's still acceptable for inclusion in 3.4 (and maybe 3.3? not sure if there'll be another bugfix release). -- versions: +Python 3.4 Added file: http://bugs.python.org/file34683/issue20375.diff ___ Python tracker <http://bugs.python.org/issue20375> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Changes by Nikolaus Rath : Added file: http://bugs.python.org/file34646/issue21057.diff ___ Python tracker <http://bugs.python.org/issue21057> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Changes by Nikolaus Rath : Removed file: http://bugs.python.org/file34645/issue21057.diff ___ Python tracker <http://bugs.python.org/issue21057> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Nikolaus Rath added the comment: I'm attaching a patch that enables TextIOWrapper to work with bytes-like objects from the underlying file descriptor. The code changes are pretty small, without introducing any significant additional complexity. For streams providing bytes objects, this patch does not change anything. For streams that are able to provide bytearrays or memoryviews, this patch removes the overhead that would be incurred by explicitly converting to bytes. -- keywords: +patch Added file: http://bugs.python.org/file34645/issue21057.diff ___ Python tracker <http://bugs.python.org/issue21057> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Nikolaus Rath added the comment: As an alternative, I have attached a pure docpatch that just documents the future behavior. Someone with commit privileges: please take your pick :-). -- Added file: http://bugs.python.org/file34633/docpatch.diff ___ Python tracker <http://bugs.python.org/issue20951> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Changes by Nikolaus Rath : -- nosy: +benjamin.peterson, hynek, stutzbach ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20578] BufferedIOBase.readinto1 is missing
Nikolaus Rath added the comment: I have attached a patch that adds readinto1() to BufferedReader and BufferedRWPair. An example use case for this method is receiving a large stream over a protocol like HTTP. You want to use a buffered reader so you can efficiently parse the header, but after that you want to stream the data as it comes in, i.e. you want to use read1 or, for improved performance, readinto1. Feedback is welcome. -- keywords: +patch Added file: http://bugs.python.org/file34632/issue20578.diff ___ Python tracker <http://bugs.python.org/issue20578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21057] TextIOWrapper does not support reading bytearrays or memoryviews
Nikolaus Rath added the comment: On 03/26/2014 03:43 AM, STINNER Victor wrote: >> class MyByteStream(BytesIO): >>def read1(self, len_): >>return memoryview(super().read(len_)) >> bs = MyByteStream(b'some data in ascii\n') > > I guess that you are trying to implement a zero-copy I/O. The problem is that > BytesIO does copy data. Right on the first count, but wrong on the second. The class I'm concerned with wants to do zero-copy I/O, but is not related to BytesIO. I only picked that to produce a minimal example. -- ___ Python tracker <http://bugs.python.org/issue21057> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
