Re: [rust-dev] AST transforming syntax extension
> Originally I *did* want to operate on AST, because I couldn't get macro
> system to parse more than the first statement when I was writing my rules
> like this:
> ( $head:stmt ; $($rest:*stmt*);+ ) => ( $head ; stmt_list!(
> $($rest);+ ) );
> Apparently interpolated statements cannot be re-parsed again?
>
Correct.
>
> Is tt specifier documented somewhere? I am guessing it stands for "token
> tree", but what is a token tree? How is it different from AST?
>
A token tree is "almost unparsed": it consists of a sequence of tokens
surrounded by `()`, `[]`, or `{}`, or a single token. It's the AST node
that the macro parser knows how to parse into actually useful AST nodes. It
isn't documented, sadly.
>
>
>> macro_rules! stmt_list(
>> ( while $cond:expr { $($body:tt)+ } ) =>
>> ( while $cond { stmt_list!( $($body)+ ) } );
>> ( $head:stmt ; $($rest:tt)+ ) =>
>> ( $head ; stmt_list!( $($rest)+ ) );
>> ( $head:stmt ) =>
>> ( $head );
>> )
>>
> That doesn't seem to work :-(
>
Could you try wrapping `{}` inside the parens on the 2nd RHS, and adding a
semicolon after all three invocations of `stmt_list!`? For future debugging
back-and-forth, please contact me in #rust (I'm `pauls`), since that'll be
easier than email?
Paul
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] AST transforming syntax extension
On Thu, Sep 19, 2013 at 12:05 PM, Paul Stansifer
wrote:
> (note that you should wrap your second RHS in `{}` in order to avoid
> hitting the following bugs: https://github.com/mozilla/rust/issues/8012
> https://github.com/mozilla/rust/issues/4375)
>
> I had misunderstood your original question: now I see that, by "operate
> on", you meant that you wanted to run the macro parser on code that has
> already been parsed from tokens into AST.
>
Originally I *did* want to operate on AST, because I couldn't get macro
system to parse more than the first statement when I was writing my rules
like this:
( $head:stmt ; $($rest:*stmt*);+ ) => ( $head ; stmt_list!( $($rest);+
) );
Apparently interpolated statements cannot be re-parsed again?
So I figured that maybe I could let Rust parser build an AST for me, and
then tweak that. Hence my question about syntax extensions.
However, later I'd discovered your lambda calculus
interpreter,
and decided to give built-in macros another go, this time using "tt" to
capture tails of statement lists.
> There isn't a mechanism to do this in the Rust macro system. However, you
> have discovered a potential workaround: capture the tokens as a token tree
> and pass them as an argument to a macro (which, after all, accepts token
> trees as arguments).
>
> The parse error is caused by the fact that (perhaps unintuitively), `tt`
> only matches a single token if it doesn't see a `(`, `[`, or `{`. I have
> not tested the following code (I fear it may run afoul of macro parser
> limitations), but this may work:
>
Is tt specifier documented somewhere? I am guessing it stands for "token
tree", but what is a token tree? How is it different from AST?
> macro_rules! stmt_list(
> ( while $cond:expr { $($body:tt)+ } ) =>
> ( while $cond { stmt_list!( $($body)+ ) } );
> ( $head:stmt ; $($rest:tt)+ ) =>
> ( $head ; stmt_list!( $($rest)+ ) );
> ( $head:stmt ) =>
> ( $head );
> )
>
That doesn't seem to work :-(
stmt_list! { let mut x = 0 ; while x < 10 { let y = x + 1 ; x = y } }
parse.rs:17:11: 17:15 error: unexpected token: `an interpolated statement`
parse.rs:17 ( $head ; stmt_list!( $($rest)+ ) );
^~~~
Vadim
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] AST transforming syntax extension
(note that you should wrap your second RHS in `{}` in order to avoid
hitting the following bugs: https://github.com/mozilla/rust/issues/8012
https://github.com/mozilla/rust/issues/4375)
I had misunderstood your original question: now I see that, by "operate
on", you meant that you wanted to run the macro parser on code that has
already been parsed from tokens into AST. There isn't a mechanism to do
this in the Rust macro system. However, you have discovered a potential
workaround: capture the tokens as a token tree and pass them as an argument
to a macro (which, after all, accepts token trees as arguments).
The parse error is caused by the fact that (perhaps unintuitively), `tt`
only matches a single token if it doesn't see a `(`, `[`, or `{`. I have
not tested the following code (I fear it may run afoul of macro parser
limitations), but this may work:
macro_rules! stmt_list(
( while $cond:expr { $($body:tt)+ } ) =>
( while $cond { stmt_list!( $($body)+ ) } );
( $head:stmt ; $($rest:tt)+ ) =>
( $head ; stmt_list!( $($rest)+ ) );
( $head:stmt ) =>
( $head );
)
Paul
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] AST transforming syntax extension
I'd love to be able to use just macros, however so far my attempts met with
little success.
Here's what I'm trying to do (right now expansion just echoes the input):
macro_rules! stmt_list(
( while $cond:expr { $body:tt } ) =>
( while $cond { stmt_list!( $body ) } );
( $head:stmt ; $rest:tt ) =>
( $head ; stmt_list!( $rest ) );
( $head:stmt ) =>
( $head );
)
fn main()
{
trace_macros!(true);
stmt_list!(
let mut x = 0;
while x < 10 {
let y = x + 1;
x = y
}
)
}
C:\rust\test>rustc --pretty expanded parse.rs
stmt_list! { let mut x = 0 ; while x < 10 { let y = x + 1 ; x = y } }
parse.rs:18:14: 18:15 error: No rules expected the token: x
parse.rs:18 while x < 10 {
^
Am I totally off-base trying to do this with macros? Do I need a syntax
extension?
Vadim
On Wed, Sep 18, 2013 at 8:06 PM, Paul Stansifer wrote:
> What you describe is the status quo for syntax extensions and macros.
>
> The standard way for a syntax extension (built into the Rust compiler, not
> a macro) to operate is to use the macro parser (which in turn uses the Rust
> parser) to turn token trees into AST nodes. See libsyntax/ext/ for
> implementations of existing syntax extensions.
>
> If you want to write a macro, on the other hand, which is sufficient for
> most needs, then you don't need to do anything, because this is what macros
> already do -- as soon as they parse, say, `$s1:stmt`, `s1` is a statement
> AST node, ready for interpolation.
>
> Does this help?
>
> Paul
>
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] AST transforming syntax extension
What you describe is the status quo for syntax extensions and macros. The standard way for a syntax extension (built into the Rust compiler, not a macro) to operate is to use the macro parser (which in turn uses the Rust parser) to turn token trees into AST nodes. See libsyntax/ext/ for implementations of existing syntax extensions. If you want to write a macro, on the other hand, which is sufficient for most needs, then you don't need to do anything, because this is what macros already do -- as soon as they parse, say, `$s1:stmt`, `s1` is a statement AST node, ready for interpolation. Does this help? Paul ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] AST transforming syntax extension
Such is not possible at the moment. Syntax extensions operate on token
trees, not AST. Maybe you'd be able to parse the input? Not sure, not
my knowledge area.
On Wed, Sep 18, 2013 at 7:54 PM, Vadim wrote:
> Hi,
> I want to write a syntax extension that operates on Rust AST, not on
> individual tokens. For example:
>
> fn foo() {
> ...
> my_ext!{
> ;
> ;
> ...
> }
> ...
> }
> where are normal Rust statements.
>
> I want to get the AST for code enclosed inside my extension block, transform
> it, and insert in place of the original node(s).
>
> Any pointers to where to start? Are there any existing extensions that do
> something close to what I want?
>
> Vadim
>
> ___
> Rust-dev mailing list
> Rust-dev@mozilla.org
> https://mail.mozilla.org/listinfo/rust-dev
>
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
