[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Łukasz Langa : -- Removed message: https://bugs.python.org/msg374253 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Batuhan Taskaya : -- nosy: +BTaskaya ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Lysandros Nikolaou : -- nosy: +lys.nikolaou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Edward K Ream added the comment: Hello all, This is a "sideways" response to this issue. I have been dithering about whether to give you a heads up. I hope you won't mind... I have just announced the leoAst.py on python-announce-list. You can read the announcement here:

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Jimmy Lai added the comment: Just found Guido mentioned LibCST. Here is a quick overview: 1. LibCST is an open source Python concrete syntax tree parser. It provides a CST looks like and feel like AST. 2. It's built by Instagram for linter and refactoring tools (exact use cases what Łukasz

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Guido van Rossum added the comment: If people are looking for a concrete CST that works now, maybe LibCST will work? https://github.com/Instagram/LibCST -- ___ Python tracker

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by kernc : -- nosy: +kernc ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Niklas Rosenstein added the comment: Lukasz, have you created a 3rd party package branching off lib2to3? I'm working on a project that is based on it (in a similar usecase as YAPF and Black) and was hoping that there may be some version maintained distinctly from the Python release

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Ivan Levkivskyi : -- nosy: +levkivskyi ___ Python tracker ___ ___

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Jakub Wilk : -- nosy: +jwilk ___ Python tracker ___ ___ Python-bugs-list mailing

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Guido van Rossum added the comment: I think merging the tokenizers still makes sense. We can then document top-level tokenize.py (in 3.8 and later) as guaranteed to be able to tokenize anything going back to Python 2.7. And since lib2to3/pgen2 it is undocumented I presume

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Łukasz Langa added the comment: [njs] > "there's a bug in handling this python 2 code, so black won't be able to > reformat it until the next major python release" Nah, we're still allowed to fix bugs in micro releases. We should have more of those instead of sitting on

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Ethan Smith : -- nosy: +Ethan Smith ___ Python tracker ___ ___ Python-bugs-list

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Zsolt Dollenstein : -- nosy: +zsol ___ Python tracker ___ ___

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Guido van Rossum added the comment: Lukasz, pleased consider seriously to move to a 3rd party package. Even pgen2. On Mon, Apr 23, 2018, 21:12 Benjamin Peterson wrote: > > Benjamin Peterson added the comment: > > The stdlib is a

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Benjamin Peterson added the comment: The stdlib is a bad place for anything that needs to evolve at a non-glacial place. For example, even when 2to3 had not yet fallen out of favor, there were effectively 3 versions of it: one 2.7 and two in maintained 3.x branches. That

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Łukasz Langa : -- keywords: +patch pull_requests: +6285 stage: -> patch review ___ Python tracker ___

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Nathaniel Smith added the comment: It does seem like it'd be unfortunate to end up in a situation like "sorry, there's a bug in handling this python 2 code, so black won't be able to reformat it until the next major python release". And I assume this issue is motivated by

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Łukasz Langa added the comment: > But lib2to3 is proof that the stdlib is just as much subject to stalling. The issue here is internal visibility. "lib2to3" is a library that supports "2to3" which is rather neglected internally since we started promoting `six` as a better

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Guido van Rossum added the comment: But lib2to3 is proof that the stdlib is just as much subject to stalling. Maybe lib2to3 and pgen2 would have a livelier future if they weren't limited to updates in sync with Python releases. --

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Łukasz Langa added the comment: > The important part is that it is maintained and kept up to date with future > language grammar changes while maintaining "backwards grammar compatibility". Yes, which is why I have trouble believing this can be effectively outsourced.

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Gregory P. Smith added the comment: +1 in general to this work. Łukasz is effectively preaching to the choir by looping me in here. :) It is a big challenge to practically support Python in that we have no good ability to parse and understand all language syntax versions via

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Łukasz Langa added the comment: > These modification are applied only before bytecodecode generation. The AST > presented to user is not modified. This bit me when implementing PEP 563 but I was then on the compile path, right. Still, the latest docstring folding would

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Łukasz Langa added the comment: > I'm in favor of unifying the tokenizers and of updating and moving pgen2 > (though I don't have time to do the work). I'm willing to do all the work as long as I have somebody to review it. Case in point: BPO-8. > Also I think you may

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Serhiy Storchaka added the comment: > - the built-in AST increasingly modifies the tree before presenting it to user > code (constant folding moved to the AST in Python 3.7); These modification are applied only before bytecodecode generation. The AST presented

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Guido van Rossum added the comment: I'm glad you've rediscovered pgen2! I'm in favor of unifying the tokenizers and of updating and moving pgen2 (though I don't have time to do the work). I'm not sure if it's technically possible to give tokenize.py the ability to tokenize

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Łukasz Langa : -- pull_requests: -6270 ___ Python tracker ___ ___

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Łukasz Langa : -- keywords: -patch ___ Python tracker ___ ___ Python-bugs-list

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Łukasz Langa added the comment: See BPO-8 for an implementation of Step 1. -- stage: patch review -> ___ Python tracker ___

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

Change by Łukasz Langa : -- keywords: +patch pull_requests: +6270 stage: -> patch review ___ Python tracker ___

[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

New submission from Łukasz Langa : Python includes a set of batteries that enable parsing of Python code. This includes its own AST (provided in the standard library under the `ast` module), as well as a pure Python tokenizer (provided in the standard library under `tokenize`