[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: Fixed again. Thanks Stefan for finding this! I apologize for the bug. -- stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: New changeset e653369e76d7da6bcbcf1f09a141f47fb77df6c3 by Guido van Rossum in branch '3.8': [3.8] bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags (#21023)

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: New changeset 2a1ee1d970460047bd88da9638f8c1789431d9ab by Guido van Rossum in branch '3.9': [3.9] bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags (#21022)

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: New changeset 9d197c7d48147a9ea2f7f7be917f35514a16524b by Guido van Rossum in branch 'master': bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags (#21021) https://github.com/python/cpython/commit/9d197c7d48147a9ea2f7f7be917f35514a16524b

[issue35975] Put back the ability to parse files where async/await aren't keywords

Stefan Behnel added the comment: Feel free to use the patch in any way you like. -- ___ Python tracker ___ ___ Python-bugs-list

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: Okay, I will add that diff (I'll probably add async def to the set of features tested). Do you mind if I just incorporate it in my diff? I could add an acknowledgment in the commit message. Otherwise you could send a separate PR. --

[issue35975] Put back the ability to parse files where async/await aren't keywords

Stefan Behnel added the comment: PRs look good to me. Here is a test that fails for me with the master branch and works with your fixes. It has a slightly odd place in the subinterpreter tests, but I would argue that it's not entirely misplaced there. You would want to know when

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Guido van Rossum : -- pull_requests: +20195 pull_request: https://github.com/python/cpython/pull/21023 ___ Python tracker ___

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Guido van Rossum : -- pull_requests: +20194 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/21022 ___ Python tracker ___

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: Hm, the backports have to be done manually, since the 3.8 parser and the 3.10 parser don't overlap, and 3.9 has both versions... -- ___ Python tracker

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Guido van Rossum : -- status: closed -> open ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: I'm fixing it -- can you verify? I have no easy way to test this. DO you think we should also add a note to the docs that this was broken in 3.8.0-3? -- ___ Python tracker

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Guido van Rossum : -- pull_requests: +20193 pull_request: https://github.com/python/cpython/pull/21021 ___ Python tracker ___

[issue35975] Put back the ability to parse files where async/await aren't keywords

Stefan Behnel added the comment: I wasn't sure which is better – solve it or leave it. But it seems a) easy to adapt to on user side, and b) solvable in CPython in a way that allows code that wants to adapt to work across 3.8.x versions. Only code that does not get adapted would risk failure

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: But it's a bug, right? The intention for sure is that the cf_feature_version field is only used when the PyCF_ONLY_AST flags is set in cf_flags, per the docs you cite as [3]. Why wouldn't it be possible to fix that? -- status: closed -> open

[issue35975] Put back the ability to parse files where async/await aren't keywords

Stefan Behnel added the comment: I was made aware [1] that the addition of the "cf_feature_version" field broke the backwards compatibility of the "PyRun_StringFlags()" function [2]. Unlike what the docs of "PyCompilerFlags" [3] suggest, the new field is not ignored there but must now be

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: Sorry for the oversight. Do you need help fixing that? -- ___ Python tracker ___ ___

[issue35975] Put back the ability to parse files where async/await aren't keywords

Petr Viktorin added the comment: It looks like this caused bpo-37072: PyAST_FromNodeObject doesn't ignore flags any more, and when PyNode_Compile passes NULL flags, it crashes. -- nosy: +petr.viktorin ___ Python tracker

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Guido van Rossum : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue35975] Put back the ability to parse files where async/await aren't keywords

miss-islington added the comment: New changeset 495da292255b92dd73758fdd0e4c7d27d82b1e57 by Miss Islington (bot) (Guido van Rossum) in branch 'master': bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: Everyone watching, the PR is now ready for review! -- ___ Python tracker ___ ___

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by STINNER Victor : -- nosy: +vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: > Would not be simpler to just drop the support of Python versions <3.7 in new > MyPy versions? Not really -- mypy has a lot of users who run it over (very) large code bases that can't easily be upgraded. Basically mypy has to support all Python versions

[issue35975] Put back the ability to parse files where async/await aren't keywords

Serhiy Storchaka added the comment: Would not be simpler to just drop the support of Python versions <3.7 in new MyPy versions? -- nosy: +serhiy.storchaka ___ Python tracker

[issue35975] Put back the ability to parse files where async/await aren't keywords

Yury Selivanov added the comment: > And the same tokenizer trick that detects 'async def' could detect 'async > for' easily. Exactly. I just never thought that we could support async generator expressions with the kind of hacks in tokenizer we had in 3.5-3.6. FWIW I still believe that

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: And the same tokenizer trick that detects 'async def' could detect 'async for' easily. See https://github.com/python/cpython/pull/12086/files#diff-30b8266a4285de981f8b1b82a8cc6231R1418 -- ___ Python tracker

[issue35975] Put back the ability to parse files where async/await aren't keywords

Yury Selivanov added the comment: > But why would we? I already have a working solution. I've heard some complaints that it's hard to migrate to Python 3.7 because async and await are keywords (although I think by now all popular libraries have migrated already), so in case you ever

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: But why would we? I already have a working solution. (Literally reverting Jelle's PR won't work anyway, because Serhiy changed how regen-token works.) -- ___ Python tracker

[issue35975] Put back the ability to parse files where async/await aren't keywords

Yury Selivanov added the comment: Actually, I think we could revert to old lexer hacks in 3.8 if we modify them to treat 'async for' tocken sequence as one meta-tocken. -- ___ Python tracker

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Guido van Rossum : -- keywords: +patch pull_requests: +12096 stage: needs patch -> patch review ___ Python tracker ___

[issue35975] Put back the ability to parse files where async/await aren't keywords

Guido van Rossum added the comment: That should still work. The strategy is as follows: - For Python 3.7 or higher, 'async' and 'await' are *always* recognized. - For Python 3.5 and 3.6, the hacks from the PEP are used. -- ___ Python tracker

[issue35975] Put back the ability to parse files where async/await aren't keywords

Yury Selivanov added the comment: Well, it's not just rolling back async/await from being keywords. Since 3.7 it's possible to create async generator expressions in non-async functions. This wasn't possible to do with old hacks on the lexer. So if you want to revert the change you risk

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Michael Sullivan : -- nosy: +msullivan ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Mayank Asthana : -- nosy: +masthana ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Guido van Rossum : -- assignee: -> gvanrossum components: +Interpreter Core stage: -> needs patch type: -> behavior versions: +Python 3.8 ___ Python tracker ___

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Ethan Smith : -- nosy: +Ethan Smith ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35975] Put back the ability to parse files where async/await aren't keywords

Change by Ivan Levkivskyi : -- nosy: +levkivskyi ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35975] Put back the ability to parse files where async/await aren't keywords

New submission from Guido van Rossum : Now that the ast module can be used to parse type comments, there’s one more feature I’d like to lobby for – a flag that modifies the grammar slightly so it resembles an older version of Python (going back to 3.4). This is used in mypy to decouple the