[issue13342] input() builtin always uses "strict" error handler

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Committed. I hope the test won't disturb the buildbots.

--
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 421c8e291221 by Antoine Pitrou in branch '3.2':
Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
http://hg.python.org/cpython/rev/421c8e291221

New changeset 992ba03d60a8 by Antoine Pitrou in branch 'default':
Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
http://hg.python.org/cpython/rev/992ba03d60a8

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> I can make it work at the interpreter prompt with your patch applied.
> Sorry for cluttering up the ticket. ;-)

That's ok, thanks a lot for testing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Stefan Holek]

Stefan Holek  added the comment:

I can make it work at the interpreter prompt with your patch applied. Sorry for 
cluttering up the ticket. ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Stefan Holek]

Stefan Holek  added the comment:

Oops, the last one wasn't meant for the bug tracker. 

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Stefan Holek]

Stefan Holek  added the comment:

Python 3.2.2 (default, Nov  4 2011, 22:28:55) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, io
>>> w = io.TextIOWrapper(sys.stdin.detach(), 'ascii', 'replace')
>>> input

>>> input()
Traceback (most recent call last):
  File "", line 1, in 
ValueError: underlying buffer has been detached

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Stefan Holek]

Stefan Holek  added the comment:

This is with Python 3.2.2 on Mac OS X 10.6 (SL). I have built Python from 
source with: ./configure; make; make install.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> However when I try your example I get
> 
> sys.stdin = io.TextIOWrapper(
> sys.stdin.detach(), 'ascii', 'replace')
> ValueError: underlying buffer has been detached

Which version of Python (and which OS?). It works fine here on latest
3.2 and 3.3 branches.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[stefanholek]

stefanholek  added the comment:

Thank you Antoine, this looks good.

However when I try your example I get

sys.stdin = io.TextIOWrapper(
sys.stdin.detach(), 'ascii', 'replace')
ValueError: underlying buffer has been detached



--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Charles-François Natali]

Charles-François Natali  added the comment:

> The bugfix itself is quite pedestrian, but the test is more interesting.

Indeed. Looks good to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Here is a patch. The bugfix itself is quite pedestrian, but the test is more 
interesting. I did what I could to fork a subprocess into a pseudoterminal so 
as to trigger the GNU readline code path. The only limitation I've found is 
that I'm unable to read further on the child's stdout after input() has been 
called. The test therefore uses a pipe to do the return checking.

--
keywords: +patch
nosy: +neologix
stage: needs patch -> patch review
Added file: http://bugs.python.org/file23610/input_readline.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> There cannot be a reason input() should be confined to "strict", or can 
> there? ;-)

Actually, there's a good reason: in the non-interactive case, input() simply 
calls sys.stdin.read(), which doesn't have encoding or errors attributes. You 
want to override sys.stdin so that it has the right error handler.

However, there is a bug in input() in that it ignores sys.stdin's error handler 
in interactive mode (where it delegates to the readline library, if present):

>>> import sys, io
>>> sys.stdin = io.TextIOWrapper(sys.stdin.detach(), "ascii", "replace")
>>> sys.stdin.read()
héhé
'h��h��\n'
>>> input()
héhé
Traceback (most recent call last):
  File "", line 1, in 
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal 
not in range(128)


If you don't mind losing GNU readline functionality, the immediate workaround 
for you is to use sys.stdin.read() directly.

--
components: +Interpreter Core -Unicode
nosy: +pitrou
stage:  -> needs patch
versions:  -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[stefanholek]

stefanholek  added the comment:

I am not quite sure how I would write a custom, readline-using input function 
in Python (access to PyOS_Readline seems required), that's why I did it in C. 
Have an example?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[Benjamin Peterson]

Benjamin Peterson  added the comment:

There's no reason you couldn't write your own input() function in Python to do 
this.

--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13342] input() builtin always uses "strict" error handler

[stefanholek]

New submission from stefanholek :

The input builtin always uses "strict" error handling for Unicode conversions. 
This means that when I enter a latin-1 string in a utf-8 environment, input 
breaks with a UnicodeDecodeError. Now don't tell me not to do that, I have a 
valid use-case. ;-)

While "strict" may be a good default choice, it is clearly not sufficient. I 
would like to propose an optional 'errors' argument to input, similar to the 
'errors' argument the decode and encode methods have.

I have in fact implemented such an input method for my own use:
https://github.com/stefanholek/rl/blob/surrogate-input/rl/input.c

While this solves my immediate needs, the fact that my implementation is 
basically just a copy of bltinmode.input with one additional argument, makes me 
think that this could be fixed in Python proper.

There cannot be a reason input() should be confined to "strict", or can there? 
;-)

--
components: Unicode
messages: 147005
nosy: ezio.melotti, stefanholek
priority: normal
severity: normal
status: open
title: input() builtin always uses "strict" error handler
type: behavior
versions: Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com