With lib2to3 going away (https://bugs.python.org/issue40360), it seems to
me that some of its functionality for handling "whitespace" can be fairly
easily added to the ast module. (By "whitespace", I mean things like
indent, dedent, comments, backslash; and also the ability to manipulate the
encoded bytes in the original source.)
Off the top of my head, I know of the following projects that use lib2to3
or similar to access the "whitespace" in the parse tree and will need a new
solution:
yapf, black, mypy, pytype, kythe.
(If they don't use lib2to3, they need to maintain a custom parser that
changes with each release of Python, so my proposal would potentially help
them.)
Are there other projects that need access to the parse tree and
"whitespace"?
I propose implementing an optional pass over the parse tree that records
lib2to3's "prefix" with each leaf node. The interface would be something
like:
def detect_encoding(source: bytes) -> str
def parse_with_whitespace(source: bytes, encoding: str, filename: str) ->
ast.Module
def unparse_bytes(tree: ast.Module) -> bytes
def unparse_str(tree: ast.Module) -> str
# Various convenience functions/properties, similar to
pytree.next_sibling etc.
parse_with_whitespace() calls ast.parse(), then does a pass over the parse
tree, adding to the leaf nodes:
prefix: str # whitespace and comments preceding token in the input
pieces: List # see below
col_byte_offset: int # start byte offset within line
src_byte_offset: int # start byte offset within source
The "pieces" field is intended to handle things like:
x = 'abc' \
"def"
y = (f'abc{x}' # comment
"def")
Each "piece" would include:
- byte offset from the beginning of the token (negative for a prefix
piece)
- detailed type (single-quote string, double-quote string, format-string,
int, float, etc.)
- source bytes
- decoded value (Unicode str)
The ast.Module class would also be extended with additional attributes that
apply to the entire source, such as the encoding.
All of this is quite fiddly, but I already have some code for dealing with
the conversions between byte and string offsets, so I don't anticipate a
huge amount of work. (The design is also inherently a bit inefficient; but
I don't want to get involved with the internals of compile().)
A related item: "Parser module in the stdlib":
https://mail.python.org/archives/list/python-...@python.org/thread/RHZ6JOEXJSQUPLJECCBEIYWWWSB3IG3O/#UJ5YEMSPSUGQFDKCIEMVALVB4WP6NYI5
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/X2HJ6I6XLIGRZDB27HRHIVQC3RXNZAY4/
Code of Conduct: http://python.org/psf/codeofconduct/
