How do I tell incomplete input from invalid input?
Hi,
I have been trying to figure out a reliable way to determine
incomplete Python script
input using Python C API. (Apology if it is OT here, I'm not sure where my post
belongs, perhaps to cplusplus-sig list.)
Apparently, most pointers lead to the Python FAQ [1] question:
How do I tell incomplete input from invalid input?
Unfortunately, this FAQ is either old or incomplete thus incorrect.
First, the proposed testcomplete() function uses internal symbols
which are not available to Python C API users. So, whoever wrote that FAQ
should be given 20 lashes with a short piece of string [2].
The second solution is incomplete or incorrect. It does not handle correctly
multi-line input longer than two lines with more flow control statements.
For example:
##
n = 10
if n 0:
...if n 100:
File stdin, line 2
if n 100:
^
IndentationError: expected an indented block
##
or
##
for n in range(0, 5):
... if n 2:
File stdin, line 2
if n 2:
^
IndentationError: expected an indented block
##
I have attached a slightly modified C++ version of the second program
from the FAQ question [1],
file faq_incomplete_input.cpp which is also available from my GitHub repo [3]
In this program, I added several FIX comments with proposed corrections.
The idea is to additionally check for
PyErr_ExceptionMatches (PyExc_IndentationError)
and
strcmp (msg, expected an indented block)
and
prompt is sys.ps2, means more code expected.
And, ignore errors until user confirms the input is finished,
so the whole input is eventually sent to the Py_CompileString
and then all exceptions are not ignored, but considered
as real result of compilation.
I simply wanted to achieve similar semantic to codeop._maybe_compile()
(called by codeop.compile_command) which performs some sort of dirty
hack in the following line:
if not code1 and repr(err1) == repr(err2):
So, the test in action for multi-line multi-statement input gives:
##
c = codeop.compile_command(for n in range(0, 3):, test, single)
err1 SyntaxError('unexpected EOF while parsing', ('test', 1, 22, 'for
n in range(0, 3):\n'))
err2 IndentationError('expected an indented block', ('test', 2, 1, '\n'))
comparison.err1 SyntaxError('unexpected EOF while parsing', ('test',
1, 22, 'for n in range(0, 3):\n'))
comparison.err2 IndentationError('expected an indented block',
('test', 2, 1, '\n'))
code None
code1 None
c = codeop.compile_command(for n in range(0, 3):\n\tif n 0:, test,
single)
err1 IndentationError('expected an indented block', ('test', 2, 11,
'\tif n 0:\n'))
err2 IndentationError('expected an indented block', ('test', 3, 1, '\n'))
comparison.err1 IndentationError('expected an indented block',
('test', 2, 11, '\tif n 0:\n'))
comparison.err2 IndentationError('expected an indented block',
('test', 3, 1, '\n'))
code None
code1 None
##
So, I reckon it make sense to use the same logic to when calling
Py_CompileString.
Does it sound as reasonable solution?
Basically, there seem to be no canonical solution anywhere presented
on how to perform incomplete input tests in reliable manner, how to perform
parsing/compilation in subsequent steps against Python code given line-by-line.
The C API used by Python function compile() is not publicly available.
There is PyRun_InteractiveLoop mechanism but it is tightly coupled to
FILE-based I/O which is not always available when Python is embedded,
so the loop is useless in number of situations.
Have I overlooked any other obvious solution?
Finally, it would be helpful if the Python FAQ is up to date.
[1]
http://docs.python.org/py3k/faq/extending.html#how-do-i-tell-incomplete-input-from-invalid-input
[2] http://mail.python.org/pipermail/python-list/2004-August/887195.html
[3] https://github.com/mloskot/workshop/blob/master/python/
Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
//
// A quick and dirty C++ version of the C program presented in Python FAQ:
// http://docs.python.org/py3k/faq/extending.html#how-do-i-tell-incomplete-input-from-invalid-input
// Modifications:
// - do not use readline library, but iostream
//
// Tested using Visual C++ 2010 (10.0) and Python 3.2 (custom Debug build)
//
// The incomplete input solution presented in the FAQ is incomplete and it does
// not allow multi-line scripts with more than 2 lines of flow control statements:
//
// n = 10
// if n 0:
//... if n 100:
// File stdin, line 2
//if n 100:
// ^
//IndentationError: expected an indented block
//
//
// n = 10
// if n 0:
//... if n
Re: How do I tell incomplete input from invalid input?
On 1/11/2012 8:50 AM, Mateusz Loskot wrote: Unfortunately, this FAQ is either old or incomplete thus incorrect. If you have a suggested change to the current text, please submit it to the tracker at bugs.python.org -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: How do I tell incomplete input from invalid input?
Terry Reedy wrote On 1/11/2012 8:50 AM, Mateusz Loskot wrote: Unfortunately, this FAQ is either old or incomplete thus incorrect. If you have a suggested change to the current text, please submit it to the tracker at bugs.python.org Yes, this is quite obvious procedure to me, but first I wanted to discuss the problem and to try to find proper solution worth to be included in the FAQ. Best regards, - -- Mateusz Loskot http://mateusz.loskot.net -- View this message in context: http://python.6.n6.nabble.com/How-do-I-tell-incomplete-input-from-invalid-input-tp3585241p3632798.html Sent from the Python - python-list mailing list archive at Nabble.com. -- http://mail.python.org/mailman/listinfo/python-list
Re: Error in Extending/Embedding FAQ, point 16: How do I tell incomplete input from invalid input?
Hi,
I found a solution thanks to another posting on c++-sig and an answer by
Andreas Klöckner :)
Thank you, Andreas!
The thread is here:
http://mail.python.org/pipermail/c++-sig/2008-April/thread.html#13470
I would like to inform the responsible of the Python Extending/Embedding
FAQ, http://www.python.org/doc/faq/extending/ about the broken code in
the FAQ and the solution I found.
I hope this might prevent other people from the frustration I found
myself in this morning (...but unfortunately also, at least partly,
from the joy I am experiencing now, after finding the new solution :).
Does anybody know how to contact the person in charge?
Thanks,
Dietrich
PS:
Of course, I still wonder about the invalid syntax error message /
code I wrote about. But ok, I hope there will be some more adequate
error message / code some day in the future :)
On Wed, 2008-04-23 at 01:09 +0900, Dietrich Bollmann wrote:
On Wed, 2008-04-23 at 00:12 +0900, Dietrich Bollmann wrote:
The following code for example:
eins = [1,
... 2,
... 3]
is accepted without any problem by the Python shell.
When using the code from the FAQ and entering it line by line
already the second line causes a simple invalid syntax error:
eins = [1,
... 2,
File stdin, line 2
2,
^
SyntaxError: invalid syntax
By the way - isn't this error message / error code just wrong in
the given situation and therefor kind of a bug?
An end of file or incomplete input error at least would
describe the situation much better - and be a better base for
functionality which is based the error code also.
---
I also thought that I should explain a little bit more exactly,
what I am intending to do with the code based on
paragraph 16 (How do I tell incomplete input from invalid input?)
of the Extending/Embedding FAQ:
I am using Python as scripting language in an application (blender).
In order to interface this application from other programs
I programmed a python command port / command socket
for this application.
Between other clients I also wrote a shell client which connects via
the command port to the application. My goal is to make it as similar
to a normal python shell as possible - and therefor I try to also mimic
the intelligent way of the Python shell to react to Python input:
- when entering a line which is a complete input,
it is immediately evaluated by the shell and the
result is printed.
- when the last entered line is erroneous,
an error message is printed immediately
- when the input is incomplete, Python waits
for other lines to complete the input
- when the line is part of a function definition etc.
python waits until an empty line is entered
before accepting the input as complete.
My problem is to understand when an input is erroneous and
when it is incomplete - which is impossible with an error message
like invalid syntax...
So here again my question: How can I make the difference
between an incomplete and an erroneous input?
The code examples in the FAQ worked fine until now - but do not
anymore for the current Python implementation.
Thanks, Dietrich
By the way: Does anybody know who is responsible for the FAQ
and could adapt the examples to the current Python version
by changing the code / annotating it?
On Wed, 2008-04-23 at 00:12 +0900, Dietrich Bollmann wrote:
Hi,
Both code examples from paragraph 16 from the Python Extending /
Embedding FAQ - 'How do I tell incomplete input from invalid
input?'
-
(
http://www.python.org/doc/faq/extending/#how-do-i-tell-incomplete-input-from-invalid-input
) do not work with the current state of Python anymore.
In the second code example, the error message returned by Python is
checked in order to differentiate errors caused by an incomplete input
from other syntax errors:
if (PyArg_ParseTuple (val, sO, msg, obj)
!strcmp (msg, unexpected EOF while parsing)) /* E_EOF */
In the current Python version there are more error messages indicating
an
incomplete Python input and I could make the code work for a while
by adding the following strings to the condition:
/* error messages indicating an incomplete input */
if (PyArg_ParseTuple(error, sO, message, obj)
(!strcmp(message, unexpected EOF while parsing) ||
!strcmp(message, expected an indented block) ||
!strcmp(message, EOF while scanning triple-quoted
string)
)
) { /* E_EOF */
but recently there are also cases which generate error messages
which are too general to be added to this list.
The following code for example:
eins = [1,
... 2,
... 3]
is accepted without any problem by the Python shell.
When
Re: Error in Extending/Embedding FAQ, point 16: How do I tell incomplete input from invalid input?
Dietrich:
The web maintainers list is [EMAIL PROTECTED] Your message will be
held for moderation, but will be seen by the team who maintain the web site.
regards
Steve
PS: Anyone who wants to *help* maintain the web site should also email
that list. It helps if you are already known to other members of the
Python community, but it's not essential.
Dietrich Bollmann wrote:
Hi,
I found a solution thanks to another posting on c++-sig and an answer by
Andreas Klöckner :)
Thank you, Andreas!
The thread is here:
http://mail.python.org/pipermail/c++-sig/2008-April/thread.html#13470
I would like to inform the responsible of the Python Extending/Embedding
FAQ, http://www.python.org/doc/faq/extending/ about the broken code in
the FAQ and the solution I found.
I hope this might prevent other people from the frustration I found
myself in this morning (...but unfortunately also, at least partly,
from the joy I am experiencing now, after finding the new solution :).
Does anybody know how to contact the person in charge?
Thanks,
Dietrich
PS:
Of course, I still wonder about the invalid syntax error message /
code I wrote about. But ok, I hope there will be some more adequate
error message / code some day in the future :)
On Wed, 2008-04-23 at 01:09 +0900, Dietrich Bollmann wrote:
On Wed, 2008-04-23 at 00:12 +0900, Dietrich Bollmann wrote:
The following code for example:
eins = [1,
... 2,
... 3]
is accepted without any problem by the Python shell.
When using the code from the FAQ and entering it line by line
already the second line causes a simple invalid syntax error:
eins = [1,
... 2,
File stdin, line 2
2,
^
SyntaxError: invalid syntax
By the way - isn't this error message / error code just wrong in
the given situation and therefor kind of a bug?
An end of file or incomplete input error at least would
describe the situation much better - and be a better base for
functionality which is based the error code also.
---
I also thought that I should explain a little bit more exactly,
what I am intending to do with the code based on
paragraph 16 (How do I tell incomplete input from invalid input?)
of the Extending/Embedding FAQ:
I am using Python as scripting language in an application (blender).
In order to interface this application from other programs
I programmed a python command port / command socket
for this application.
Between other clients I also wrote a shell client which connects via
the command port to the application. My goal is to make it as similar
to a normal python shell as possible - and therefor I try to also mimic
the intelligent way of the Python shell to react to Python input:
- when entering a line which is a complete input,
it is immediately evaluated by the shell and the
result is printed.
- when the last entered line is erroneous,
an error message is printed immediately
- when the input is incomplete, Python waits
for other lines to complete the input
- when the line is part of a function definition etc.
python waits until an empty line is entered
before accepting the input as complete.
My problem is to understand when an input is erroneous and
when it is incomplete - which is impossible with an error message
like invalid syntax...
So here again my question: How can I make the difference
between an incomplete and an erroneous input?
The code examples in the FAQ worked fine until now - but do not
anymore for the current Python implementation.
Thanks, Dietrich
By the way: Does anybody know who is responsible for the FAQ
and could adapt the examples to the current Python version
by changing the code / annotating it?
On Wed, 2008-04-23 at 00:12 +0900, Dietrich Bollmann wrote:
Hi,
Both code examples from paragraph 16 from the Python Extending /
Embedding FAQ - 'How do I tell incomplete input from invalid
input?'
-
(
http://www.python.org/doc/faq/extending/#how-do-i-tell-incomplete-input-from-invalid-input
) do not work with the current state of Python anymore.
In the second code example, the error message returned by Python is
checked in order to differentiate errors caused by an incomplete input
from other syntax errors:
if (PyArg_ParseTuple (val, sO, msg, obj)
!strcmp (msg, unexpected EOF while parsing)) /* E_EOF */
In the current Python version there are more error messages indicating
an
incomplete Python input and I could make the code work for a while
by adding the following strings to the condition:
/* error messages indicating an incomplete input */
if (PyArg_ParseTuple(error, sO, message, obj)
(!strcmp(message, unexpected EOF while parsing) ||
!strcmp(message, expected an indented block) ||
!strcmp(message, EOF while scanning triple-quoted
string)
)
) { /* E_EOF
Error in Extending/Embedding FAQ, point 16: How do I tell incomplete input from invalid input?
Hi,
Both code examples from paragraph 16 from the Python Extending /
Embedding FAQ - 'How do I tell incomplete input from invalid input?'
-
(
http://www.python.org/doc/faq/extending/#how-do-i-tell-incomplete-input-from-invalid-input
) do not work with the current state of Python anymore.
In the second code example, the error message returned by Python is
checked in order to differentiate errors caused by an incomplete input
from other syntax errors:
if (PyArg_ParseTuple (val, sO, msg, obj)
!strcmp (msg, unexpected EOF while parsing)) /* E_EOF */
In the current Python version there are more error messages indicating an
incomplete Python input and I could make the code work for a while
by adding the following strings to the condition:
/* error messages indicating an incomplete input */
if (PyArg_ParseTuple(error, sO, message, obj)
(!strcmp(message, unexpected EOF while parsing) ||
!strcmp(message, expected an indented block) ||
!strcmp(message, EOF while scanning triple-quoted
string)
)
) { /* E_EOF */
but recently there are also cases which generate error messages
which are too general to be added to this list.
The following code for example:
eins = [1,
... 2,
... 3]
is accepted without any problem by the Python shell.
When using the code from the FAQ and entering it line by line
already the second line causes a simple invalid syntax error:
eins = [1,
... 2,
File stdin, line 2
2,
^
SyntaxError: invalid syntax
which is to general to be integrated into the list of tested
error messages as it might be caused also by code like:
one two
File stdin, line 1
one two
^
SyntaxError: invalid syntax
which generates an invalid syntax error even in the Python shell.
I also tried the first code example of paragraph
'16 How do I tell incomplete input from invalid input?'
of the FAQ in order to see if it could be used to make the
difference between syntax errors and incomplete code errors.
But - as in the case before - the returned error
code is E_SYNTAX (14 = Syntax error) and not E_EOF (11 = End Of File)
as should be expected.
Is there anybody who has an idea how to differentiate the
first case from the second in order to mimic the behaviour of
the Python shell from c code?
If this shouldn't be possible lists split into different lines
couldn't be accepted anymore or the feature of the Python shell
to described in paragraph 16 of the faq:
Sometimes you want to emulate the Python interactive interpreter's
behavior, where it gives you a continuation prompt when the input
is incomplete (e.g. you typed the start of an if statement or you
didn't close your parentheses or triple string quotes), but it gives
you a syntax error message immediately when the input is invalid.
would have to be given up and every entered line of code would have to
be terminated by an empty line before evaluation :(
Thanks for any help, Dietrich
--
http://mail.python.org/mailman/listinfo/python-list
Re: Error in Extending/Embedding FAQ, point 16: How do I tell incomplete input from invalid input?
On Wed, 2008-04-23 at 00:12 +0900, Dietrich Bollmann wrote:
The following code for example:
eins = [1,
... 2,
... 3]
is accepted without any problem by the Python shell.
When using the code from the FAQ and entering it line by line
already the second line causes a simple invalid syntax error:
eins = [1,
... 2,
File stdin, line 2
2,
^
SyntaxError: invalid syntax
By the way - isn't this error message / error code just wrong in
the given situation and therefor kind of a bug?
An end of file or incomplete input error at least would
describe the situation much better - and be a better base for
functionality which is based the error code also.
---
I also thought that I should explain a little bit more exactly,
what I am intending to do with the code based on
paragraph 16 (How do I tell incomplete input from invalid input?)
of the Extending/Embedding FAQ:
I am using Python as scripting language in an application (blender).
In order to interface this application from other programs
I programmed a python command port / command socket
for this application.
Between other clients I also wrote a shell client which connects via
the command port to the application. My goal is to make it as similar
to a normal python shell as possible - and therefor I try to also mimic
the intelligent way of the Python shell to react to Python input:
- when entering a line which is a complete input,
it is immediately evaluated by the shell and the
result is printed.
- when the last entered line is erroneous,
an error message is printed immediately
- when the input is incomplete, Python waits
for other lines to complete the input
- when the line is part of a function definition etc.
python waits until an empty line is entered
before accepting the input as complete.
My problem is to understand when an input is erroneous and
when it is incomplete - which is impossible with an error message
like invalid syntax...
So here again my question: How can I make the difference
between an incomplete and an erroneous input?
The code examples in the FAQ worked fine until now - but do not
anymore for the current Python implementation.
Thanks, Dietrich
By the way: Does anybody know who is responsible for the FAQ
and could adapt the examples to the current Python version
by changing the code / annotating it?
On Wed, 2008-04-23 at 00:12 +0900, Dietrich Bollmann wrote:
Hi,
Both code examples from paragraph 16 from the Python Extending /
Embedding FAQ - 'How do I tell incomplete input from invalid
input?'
-
(
http://www.python.org/doc/faq/extending/#how-do-i-tell-incomplete-input-from-invalid-input
) do not work with the current state of Python anymore.
In the second code example, the error message returned by Python is
checked in order to differentiate errors caused by an incomplete input
from other syntax errors:
if (PyArg_ParseTuple (val, sO, msg, obj)
!strcmp (msg, unexpected EOF while parsing)) /* E_EOF */
In the current Python version there are more error messages indicating
an
incomplete Python input and I could make the code work for a while
by adding the following strings to the condition:
/* error messages indicating an incomplete input */
if (PyArg_ParseTuple(error, sO, message, obj)
(!strcmp(message, unexpected EOF while parsing) ||
!strcmp(message, expected an indented block) ||
!strcmp(message, EOF while scanning triple-quoted
string)
)
) { /* E_EOF */
but recently there are also cases which generate error messages
which are too general to be added to this list.
The following code for example:
eins = [1,
... 2,
... 3]
is accepted without any problem by the Python shell.
When using the code from the FAQ and entering it line by line
already the second line causes a simple invalid syntax error:
eins = [1,
... 2,
File stdin, line 2
2,
^
SyntaxError: invalid syntax
which is to general to be integrated into the list of tested
error messages as it might be caused also by code like:
one two
File stdin, line 1
one two
^
SyntaxError: invalid syntax
which generates an invalid syntax error even in the Python shell.
I also tried the first code example of paragraph
'16 How do I tell incomplete input from invalid input?'
of the FAQ in order to see if it could be used to make the
difference between syntax errors and incomplete code errors.
But - as in the case before - the returned error
code is E_SYNTAX (14 = Syntax error) and not E_EOF (11 = End Of File)
as should be expected.
Is there anybody who has an idea how to differentiate the
first case from the second in order to mimic the behaviour
