Hi Benjamin,
I don't like the idea of just passing a stream of tokens to the macro
processor, for three reasons.
1. It makes life unnecessarily difficult for macro implementers by
forcing them to parse the stream of tokens.
Macros may contain difficult to parse constructs, and other macros.
If all that ends up happening is that macro processors reconstruct the
source and pass it to ast.parse(), then what's the point?
2. I want macros to have familiar syntactic structure, even if the
semantics are novel.
3. Pablo has already pointed out the potential performance impact of
exposing the AST at the Python level. Passing tokens around would mean
that both the AST nodes and lexer tokens would need to be Python
objects, making the performance impact much greater.
Cheers,
Mark.
On 16/09/2020 2:33 pm, Benjamin Peterson wrote:
On Tue, Sep 15, 2020, at 23:54, Guido van Rossum wrote:
On Tue, Sep 15, 2020 at 7:31 PM Benjamin Peterson <benja...@python.org> wrote:
On Tue, Sep 15, 2020, at 20:10, Greg Ewing wrote:
Maybe the PEP should propose an AST of its own, which would initially
be a third thing separate from either of the existing ones, with
the possibility of adopting it as the ast module representation
some time in the future.
The rust approach to this problem to pass only a token stream to the macro.
Then, the macro writer brings their own parser.
And (I looked this up [1]) it returns another token stream which is
then re-parsed by the standard parser. I'm not quite sure who is
responsible for determining where the macro ends -- the standard parser
or the macro parser.
The standard parser determines the macro boundaries and passes the macro
implementation only the relevant tokens.
A nice side effect of this would be that (hopefully) the macro syntax
doesn't need to have the ugly special cases needed to allow `import!
foo as bar` and `from! mod import foo as bar` (though these still need
to be predefined macros).
I'm not entirely sure how this would preserve the hygiene of the macros
though, since the macro's parser could just do a token-by-token
substitution without checking the context, effectively what a macro in
C does. Forcing an AST on the macros restricts macros somewhat (e.g. it
requires the input arguments to look like well-formed expressions) but
that's probably a good thing.
The rust macro TokenStream is a bit higher level than a traditional token
stream. In particular, the macro implementation can request that new
identifiers it inserts into the stream are only resolvable within the macro for
hygiene. There's a bit about that here:
https://doc.rust-lang.org/proc_macro/struct.Ident.html#impl
(Hm, or not. How would you implement something like PEP 505 using
macros? It introduces new tokens like `?.` and `??`.
Is there any proposal that would let you add new tokens?
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/GFDSVQ4Q7KBPUOKCIPGU724ANQZGVWV3/
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/6RYU5Q5UMWACBSDLNHBPINCDLW5D4KWI/
Code of Conduct: http://python.org/psf/codeofconduct/
