[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Cool! Be aware that using it outside codeop is currently unsupported:

https://github.com/python/cpython/blob/5d53cf30f9cb3758849e859db5d4602cb7c521f7/Lib/codeop.py#L43-L47

So have in mind that the flag, mechanism and semantics can change without 
previous notice

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Mateusz Loskot]

Mateusz Loskot  added the comment:

Possibly, the new partial-input mode of the parser 
(https://bugs.python.org/issue46521#msg412832) will improve this issue in 3.11. 
I'm going to play with it soon and I will report back.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset c7af838805ddf52320bce3d5978bfdd37eed1b3a by Pablo Galindo Salgado 
in branch '3.10':
[3.10] bpo-46502: Remove "How do I tell incomplete input" from FAQ (GH-30925) 
(GH-30933)
https://github.com/python/cpython/commit/c7af838805ddf52320bce3d5978bfdd37eed1b3a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset dafada393f9a790461430e2493ea1379e938b51a by Pablo Galindo Salgado 
in branch '3.9':
[3.9] bpo-46502: Remove "How do I tell incomplete input" from FAQ (GH-30925) 
(GH-30934)
https://github.com/python/cpython/commit/dafada393f9a790461430e2493ea1379e938b51a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +29113
pull_request: https://github.com/python/cpython/pull/30934

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +29112
pull_request: https://github.com/python/cpython/pull/30933

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset f0a648152f2d8011f47cc49873438ebaf01d3f82 by Mateusz Łoskot in 
branch 'main':
bpo-46502: Remove "How do I tell incomplete input" from FAQ (GH-30925)
https://github.com/python/cpython/commit/f0a648152f2d8011f47cc49873438ebaf01d3f82


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Mateusz Loskot]

Mateusz Loskot  added the comment:

Not quite the answer I'd like to received, but thank you very much for the 
explanation.

I've submitted pull request removing the deprecated FAQ entry
https://github.com/python/cpython/pull/30925

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
nosy: +python-dev
nosy_count: 4.0 -> 5.0
pull_requests: +29104
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/30925

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> However, I'm very keen to learn what is an alternative solution then to 
> distinguish hard invalid from incomplete input. IOW, what would be the new 
> way of achieving what's described in the old FAQ?

Unfortunately, there is no new way to do this from C due to how the new parser 
works. The only supported way is to manually import the codeop module from C 
(https://docs.python.org/3/library/codeop.html#module-codeop) and use its API 
as IDLE does, but this API can differ and will present its own complications, 
as unfortunately is implemented in a bit hacky way.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Mateusz Loskot]

Mateusz Loskot  added the comment:

Irit, I can and I will submit a patch.

However, I'm very keen to learn what is an alternative solution then to 
distinguish hard invalid from incomplete input. IOW, what would be the new way 
of achieving what's described in the old FAQ?

I believe it would be more useful to the community if I updated the sample in 
the FAQ instead of just removing it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Irit Katriel]

Change by Irit Katriel :


--
versions: +Python 3.11, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Irit Katriel]

Irit Katriel  added the comment:

Mateusz, would you like to submit a patch to remove this section from the faq?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Indeed, these new errors are due to the new parser. Unfortunately, the new 
parser doesn't allow to check against incomplete input the way the old one did 
so that piece of documentation is unfortunately outdated.

We should remove it to avoid further confusion.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Irit Katriel]

Irit Katriel  added the comment:

The python parser was completely rewritten in version 3.9, so it's likely that 
this caused changes to error messages.

Adding @pablogsal and @lys.nikolaou in case they can say more.

--
nosy: +iritkatriel, lys.nikolaou, pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46502] Py_CompileString no longer allows to tell "incomplete input" from "invalid input"

[Mateusz Loskot]

New submission from Mateusz Loskot :

Something has changed in Python 3.7 through 3.10 (I'm observing it in 3.10) in 
behaviour of the Python C API function Py_CompileString such that for an 
incomplete input it no longer raises

SyntaxError: "unexpected EOF while parsing"

but

IndentationError: expected an indented block after ...

The new behaviour makes the sample program from the "How do I tell “incomplete 
input” from “invalid input”?" at https://docs.python.org/3/faq/extending.html 
no longer work as described there.

For example:

```
for i in []:
```

raises

IndentationError: expected an indented block after 'for' statement on line 1


```
if True:
```

raises

IndentationError: expected an indented block after 'if' statement on line 1

instead of 

SyntaxError: unexpected EOF while parsing

This effectively makes it impossible to detect incomplete input using the 
Py_CompileString in applications where it is not possible to use 
PyRun_InteractiveLoop.

I have failed to identify what could be related changes in the release notes 
and the documentation does not seem to offer any update on that.
So, I'm assuming the new behaviour is not desired or expected.

Attached, is the VS 2022 screenshot with debugging session of the sample 
program from the FAQ presenting the difference in the behaviour between Python 
3.6 and 3.10 on Windows.

--
components: C API
files: Py_CompileString-Py36-vs-Py310.png
messages: 411484
nosy: mloskot
priority: normal
severity: normal
status: open
title: Py_CompileString no longer allows to tell "incomplete input" from 
"invalid input"
versions: Python 3.10
Added file: https://bugs.python.org/file50582/Py_CompileString-Py36-vs-Py310.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com