[issue13322] buffered read() and write() does not raise BlockingIOError

STINNER Victor added the comment: See also bpo-32561: Add API to io objects for non-blocking reads/writes. -- ___ Python tracker ___

[issue13322] buffered read() and write() does not raise BlockingIOError

Change by Jakub Stasiak : -- nosy: +jstasiak ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue13322] buffered read() and write() does not raise BlockingIOError

STINNER Victor added the comment: I closed bpo-24560 as a duplicate of this issue: codecs.StreamReader doesn't work with nonblocking streams: TypeError: can't concat bytes to NoneType. -- ___ Python tracker

[issue13322] buffered read() and write() does not raise BlockingIOError

STINNER Victor added the comment: I closed bpo-26292 as a duplicate of this issue: Raw I/O writelines() broken for non-blocking I/O. -- ___ Python tracker ___

[issue13322] buffered read() and write() does not raise BlockingIOError

STINNER Victor added the comment: I closed bpo-35762 as a duplicate of this issue: subprocess.Popen with universal_newlines and nonblocking streams fails with "can't concat NoneType to bytes". -- nosy: +vstinner ___ Python tracker

[issue13322] buffered read() and write() does not raise BlockingIOError

Martin Panter added the comment: Issue 35762 was opened specifically about Izbyshev’s case: TextIOWrapper behaviour with a non-blocking file. Calling “os.fdopen” with mode='r' (text mode) returns a TextIOWrapper object. -- ___ Python tracker

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou added the comment: Yes, I think adding a note in the docs is reasonable. -- ___ Python tracker ___

[issue13322] buffered read() and write() does not raise BlockingIOError

Alexey Izbyshev added the comment: Yes, your claim is confirmed by the fact that there have been little interest in this issue since 2011. Still, non-blocking behavior is incorrectly specified in the docs and is inconsistent (as investigated by Martin). And obscure errors

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou added the comment: Generally I doubt anyone is using the non-blocking semantics of the Python 3 I/O stack. People doing non-blocking I/O generally do it with sockets instead, which tend to reproduce quite literally the POSIX behaviour and error codes.

[issue13322] buffered read() and write() does not raise BlockingIOError

Alexey Izbyshev added the comment: For added fun: at least one part of the standard library doesn't expect None returns from read() in the buffering layer. >>> import os >>> r, w = os.pipe2(os.O_NONBLOCK) >>> f = os.fdopen(r, 'r') >>> f.read() Traceback (most recent call

[issue13322] buffered read() and write() does not raise BlockingIOError

Changes by Petri Lehtinen : -- nosy: -petri.lehtinen ___ Python tracker ___ ___

[issue13322] buffered read() and write() does not raise BlockingIOError

Changes by Evgeny Kapun : -- nosy: +abacabadabacaba ___ Python tracker ___ ___

[issue13322] buffered read() and write() does not raise BlockingIOError

Martin Panter added the comment: See also Issue 1191964, discussing inconsistencies and how to differentiate the four non-blocking special results in a new API: * read signals permanent EOF * read signals no data temporarily available * write signals a permanently broken pipe or connection *

[issue13322] buffered read() and write() does not raise BlockingIOError

Martin Panter added the comment: Looking at test_io.BufferedReaderTest.test_read_non_blocking(), at Lib/test/test_io.py:1037, there are explicit tests for ‘peek(1) == b ’ and ‘read() is None’. The peek() test was added in revision 3049ac17e256, in 2009 (large merge of “io” implementation in

[issue13322] buffered read() and write() does not raise BlockingIOError

Martin Panter added the comment: I’m adding nonblock-none.patch, which changes all the simple buffered read methods to return None when there is no non-blocking data available. I’d be interested to see if other people thought this was a sensible change. * Documented existing behaviour of C

[issue13322] buffered read() and write() does not raise BlockingIOError

Martin Panter added the comment: My experiments with buffered and unbuffered readers wrapping a non-blocking TCP socket, with no data received: Method Buffered impl. Buffered doc.SocketIO impl. RawIOBase doc. = == === == ==

[issue13322] buffered read() and write() does not raise BlockingIOError

Changes by Martin Panter vadmium...@gmail.com: -- nosy: +vadmium versions: +Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13322 ___ ___

[issue13322] buffered read() and write() does not raise BlockingIOError

Matt Joiner anacro...@gmail.com added the comment: The patches only fix write? What about read? http://bugs.python.org/issue13858 -- nosy: +anacrolix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13322

[issue13322] buffered read() and write() does not raise BlockingIOError

Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset ac2c4c62b486 by Antoine Pitrou in branch '3.2': Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is http://hg.python.org/cpython/rev/ac2c4c62b486 New changeset 3cd1985ed04f by Antoine Pitrou

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: Thanks again. Just a nit: the tests should be in MiscIOTest, since they don't directly instantiate the individual classes. Also, perhaps it would be nice to check that the exception's errno attribute is EAGAIN. Done. -- Added file:

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou pit...@free.fr added the comment: Thanks. Who should I credit? sbt? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13322 ___ ___

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: Thanks. Who should I credit? sbt? Yeah, thanks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13322 ___

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: Here is an updated patch which uses the real errno. It also gets rid of the restore_pos argument of _bufferedwriter_flush_unlocked() which is always set to false -- I guess buffered_flush_and_rewind_unlocked() is used instead. -- Added file:

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou pit...@free.fr added the comment: Thanks again. Just a nit: the tests should be in MiscIOTest, since they don't directly instantiate the individual classes. Also, perhaps it would be nice to check that the exception's errno attribute is EAGAIN. -- stage: needs patch -

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: Ouch. Were they only non-blocking codepaths? Yes. raw_pos is the position which the underlying raw stream is currently at. It only needs to be modified when a successful write(), read() or seek() is done on the raw stream. Do you mean

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou pit...@free.fr added the comment: Do you mean self-raw_pos should give the same answer as self.raw.tell()? (But that seems to be the definition of self-abs_pos.) Or is it the buffer offset which corresponds to self.raw.tell()? The latter. --

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou pit...@free.fr added the comment: Hi, Testing the patch a bit more thoroughly, I found that data received from the readable end of the pipe can be corrupted by the C implementation. This seems to be because two of the previously dormant codepaths did not properly maintain

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: Testing the patch a bit more thoroughly, I found that data received from the readable end of the pipe can be corrupted by the C implementation. This seems to be because two of the previously dormant codepaths did not properly maintain the necessary

[issue13322] buffered read() and write() does not raise BlockingIOError

Charles-François Natali neolo...@free.fr added the comment: write() is a bit simpler, since BlockingIOError has a characters_written attribute which is meant to inform you of the partial success: we can just reuse that. That said, BlockingIOError could grow a partial_read attribute

[issue13322] buffered read() and write() does not raise BlockingIOError

Charles-François Natali neolo...@free.fr added the comment: This is a hairy issue Indeed. Performing partial read/write may sound imperfect, but using buffered I/O around non-blockind FD is definitely not a good idea. Also, the advantage of the current approach is that at least, no data is

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou pit...@free.fr added the comment: Also, the advantage of the current approach is that at least, no data is ever lost But what about the buggy readline() behaviour? -- ___ Python tracker rep...@bugs.python.org

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou pit...@free.fr added the comment: Note that Java's BufferedInputStream and ReadableByteChannel also return partial reads. Apparently, they are specified to, even for blocking streams (which I find a bit weird, and the language in the docs seems deliberately vague). Python's

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: No one has suggested raising BlockingIOError and DISCARDING the data when a partial read has occurred. The docs seem to imply that the partially read data should be returned since they only say that BlockingIOError should be raised if there is

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: But what about the buggy readline() behaviour? Just tell people that if the return value is a string which does not end in '\n' then it might caused by EOF or EAGAIN. They can just call readline() again to check which. --

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: The third arg of BlockingIOError is used in two quite different ways. In write(s) it indicates the number of bytes of s which have been consumed (ie written to the raw file or buffered). But in flush() and flush_unlocked() (in _pyio) it indicates the

[issue13322] buffered read() and write() does not raise BlockingIOError

Charles-François Natali neolo...@free.fr added the comment: Apparently, they are specified to, even for blocking streams (which I find a bit weird, and the language in the docs seems deliberately vague). As an additional convenience, it attempts to read as many bytes as possible by

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: Currently a BlockingIOError exception raised by flush() sets characters_written to the number of bytes fushed from the internal buffer. This is undocument (although there is a unit test which tests for it) and causes confusion because characters_written

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou pit...@free.fr added the comment: But what about the buggy readline() behaviour? Just tell people that if the return value is a string which does not end in '\n' then it might caused by EOF or EAGAIN. They can just call readline() again to check which. Sounds

[issue13322] buffered read() and write() does not raise BlockingIOError

Charles-François Natali neolo...@free.fr added the comment: But then what's the point of using buffered I/O at all? If it can't offer anything more than raw I/O, I'd rather do something like raise a RuntimeError(buffered I/O doesn't work with non-blocking streams) when the raw stream returns

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: Another possibility would be that, since lines are usually reasonably sized, they should fit in the buffer (which is 8KB by default). So we could do the extra effort of buffering the data and return it once the line is complete: if the buffer fills

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: The attached patch makes BufferedWrite.write() raise BlockingIOError when the raw file is non-blocking and the write would block. -- keywords: +patch Added file: http://bugs.python.org/file23613/write_blockingioerror.patch

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: Wierdly, it looks like BlockingIO is not raised anywhere in the code for the C implementation of io. Even more wierdly, in the Python implementation of io, BlockingIOError is only ever raised by except clauses which have already caught

[issue13322] buffered read() and write() does not raise BlockingIOError

Antoine Pitrou pit...@free.fr added the comment: Wierdly, it looks like BlockingIO is not raised anywhere in the code for the C implementation of io. That would explain why it isn't raised :) This is a hairy issue: read(n) is documented as returning either n bytes or nothing. But what if

[issue13322] buffered read() and write() does not raise BlockingIOError

New submission from sbt shibt...@gmail.com: According to the the documentation, BufferedReader.read() and BufferedWriter.write() should raise io.BlockingIOError if the file is in non-blocking mode and the operation cannot succeed without blocking. However, BufferedReader.read() returns None

[issue13322] buffered read() and write() does not raise BlockingIOError

Changes by sbt shibt...@gmail.com: -- type: - behavior versions: +Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13322

[issue13322] buffered read() and write() does not raise BlockingIOError

Changes by Antoine Pitrou pit...@free.fr: -- components: +IO, Library (Lib) nosy: +pitrou versions: -Python 2.6, Python 3.1, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13322

[issue13322] buffered read() and write() does not raise BlockingIOError

Changes by Petri Lehtinen pe...@digip.org: -- nosy: +petri.lehtinen ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13322 ___ ___ Python-bugs-list

[issue13322] buffered read() and write() does not raise BlockingIOError

sbt shibt...@gmail.com added the comment: BufferedReader.readinto() should also raise BlockingIOError according to the docs. Updated unittest checks for that also. BTW, The documentation for BufferedIOBase.read() says that BlockingIOError should be raised if nothing can be read in