[issue28162] WindowsConsoleIO readall() fails if first line starts with Ctrl+Z

[Donald Stufft]

Changes by Donald Stufft :


--
pull_requests: +1057

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28162] WindowsConsoleIO readall() fails if first line starts with Ctrl+Z

[Steve Dower]

Steve Dower added the comment:

I didn't see your patch, but I made basically the same fix and added a test.

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28162] WindowsConsoleIO readall() fails if first line starts with Ctrl+Z

[Roundup Robot]

Roundup Robot added the comment:

New changeset 4d4aefa52f49 by Steve Dower in branch '3.6':
Issue #28162: Fixes Ctrl+Z handling in console readall()
https://hg.python.org/cpython/rev/4d4aefa52f49

New changeset 947fa496ca6f by Steve Dower in branch 'default':
Issue #28162: Fixes Ctrl+Z handling in console readall()
https://hg.python.org/cpython/rev/947fa496ca6f

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28162] WindowsConsoleIO readall() fails if first line starts with Ctrl+Z

[Eryk Sun]

Eryk Sun added the comment:

For breaking out of the readall while loop, you only need to check if the 
current read is empty:

/* when the read is empty we break */
if (n == 0)
break;

Also, the logic is wrong here:

if (len == 0 || buf[0] == '\x1a' && _buflen(self) == 0) {
/* when the result starts with ^Z we return an empty buffer */
PyMem_Free(buf);
return PyBytes_FromStringAndSize(NULL, 0);
}

This is true when len is 0 or when buf[0] is Ctrl+Z and _buflen(self) is 0. 
Since buf[0] shouldn't ever be Ctrl+Z here (low-level EOF handling is 
abstracted in read_console_w), it's never checking the internal buffer. We can 
easily see this going wrong here:

>>> a = sys.stdin.buffer.raw.read(1); b = sys.stdin.buffer.raw.read()
Ā^Z
>>> a
b'\xc4'
>>> b
b''

It misses the remaining byte in the internal buffer.

This check can be simplified as follows:

rn = _buflen(self);

if (len == 0 && rn == 0) {
/* return an empty buffer */
PyMem_Free(buf);
return PyBytes_FromStringAndSize(NULL, 0);
}

After this the code assumes that len isn't 0, which leads to more 
WideCharToMultiByte failure cases. 

In the last conversion it's overwrite bytes_size without including rn. 

I'm not sure what's going on with _PyBytes_Resize(, n * sizeof(wchar_t)). 
ISTM, it should be resized to bytes_size, and make sure this includes rn.

Finally, _copyfrombuf is repeatedly overwriting buf[0] instead of writing to 
buf[n]. 

With the attached patch, the behavior seems correct now:

>>> sys.stdin.buffer.raw.read()
^Z
b''

>>> sys.stdin.buffer.raw.read()
abc^Z
^Z
b'abc\x1a\r\n'

Split U+0100:

>>> a = sys.stdin.buffer.raw.read(1); b = sys.stdin.buffer.raw.read()
Ā^Z
>>> a
b'\xc4'
>>> b
b'\x80'

Split U+1234:

>>> a = sys.stdin.buffer.raw.read(1); b = sys.stdin.buffer.raw.read()
ሴ^Z
>>> a
b'\xe1'
>>> b
b'\x88\xb4'

The buffer still can't handle splitting an initial non-BMP character, stored as 
a surrogate pair. Both codes end up as replacement characters because they 
aren't transcoded as a unit.

Split U+0001:

>>> a = sys.stdin.buffer.raw.read(1); b = sys.stdin.buffer.raw.read()
^Z
^Z
>>> a
b'\xef'
>>> b
b'\xbf\xbd\xef\xbf\xbd\x1a\r\n'

--
keywords: +patch
status: closed -> open
Added file: http://bugs.python.org/file44766/issue_28162_01.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28162] WindowsConsoleIO readall() fails if first line starts with Ctrl+Z

[Steve Dower]

Steve Dower added the comment:

The fix here was changing "n > 0 && buf[len] == '\x1a'" into "n == 0 || 
buf[len] == '\x1a'" when returning an empty bytes. Previously we were falling 
into the conversion code which wasn't happy with n == 0.

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28162] WindowsConsoleIO readall() fails if first line starts with Ctrl+Z

[Roundup Robot]

Roundup Robot added the comment:

New changeset d0bab9fda568 by Steve Dower in branch '3.6':
Issue #28161: Opening CON for write access fails
https://hg.python.org/cpython/rev/d0bab9fda568

New changeset 187a114b9ef4 by Steve Dower in branch 'default':
Issue #28161: Opening CON for write access fails
https://hg.python.org/cpython/rev/187a114b9ef4

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28162] WindowsConsoleIO readall() fails if first line starts with Ctrl+Z

[Steve Dower]

Steve Dower added the comment:

Unfortunately, very hard to test this because it requires interacting with an 
actual console. But it's an easy fix.

--
assignee:  -> steve.dower
stage: test needed -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28162] WindowsConsoleIO readall() fails if first line starts with Ctrl+Z

[Eryk Sun]

New submission from Eryk Sun:

For a console readall(), if the first line starts with '\x1a' (i.e. Ctrl+Z), it 
breaks out of its read loop before incrementing len. Thus the input isn't 
handled properly as EOF, for which the check requires len > 0. Instead it ends 
up calling WideCharToMultiByte with len == 0, which fails as follows:

>>> sys.stdin.buffer.raw.read()
^Z
Traceback (most recent call last):
  File "", line 1, in 
OSError: [WinError 87] The parameter is incorrect

--
components: IO, Library (Lib), Windows
messages: 276508
nosy: eryksun, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
stage: test needed
status: open
title: WindowsConsoleIO readall() fails if first line starts with Ctrl+Z
type: behavior
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com