DIP 50 - AST macros
I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 -- /Jacob Carlborg
Re: DIP 50 - AST macros
Jacob Carlborg: http://wiki.dlang.org/DIP50 I suggest to add some more use cases (possibly with their implementation). Bye, bearophile
Re: DIP 50 - AST macros
On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote: Jacob Carlborg: http://wiki.dlang.org/DIP50 I suggest to add some more use cases (possibly with their implementation). Bye, bearophile A scalable and elegant solution to the "inspection" problem may be a AST use case. Discussed here: http://forum.dlang.org/thread/qconpedgdkyeawmdz...@forum.dlang.org --rt
Re: DIP 50 - AST macros
<[ .. ]> syntax looks very foreign. How about m{...} (with q{...} as a
precedent)?
On Sun, Nov 10, 2013 at 5:08 PM, Rob T wrote:
> On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote:
>
>> Jacob Carlborg:
>>
>> http://wiki.dlang.org/DIP50
>>>
>>
>> I suggest to add some more use cases (possibly with their implementation).
>>
>> Bye,
>> bearophile
>>
>
> A scalable and elegant solution to the "inspection" problem may be a AST
> use case.
>
> Discussed here:
> http://forum.dlang.org/thread/qconpedgdkyeawmdz...@forum.dlang.org
>
> --rt
>
Re: DIP 50 - AST macros
Jacob Carlborg: http://wiki.dlang.org/DIP50 It's also useful to take a look at what F# is doing: http://tomasp.net/blog/2013/computation-zoo-padl/index.html Bye, bearophile
Re: DIP 50 - AST macros
People have shunned proposals to have @mixin functions because it wouldn't be obvious at call site that some statement is executed under a mixin (which could access all variables in scope etc). The same will happen here; I think it should be clear at call site that a macro is used. How about: macro!myAssert(1 + 2 == 4); instead of myAssert(1 + 2 == 4); On Sun, Nov 10, 2013 at 1:20 PM, Jacob Carlborg wrote: > I've been thinking quite long of how AST macros could look like in D. I've > been posting my vision of AST macros here in the newsgroup a couple of > times already. I've now been asked to create a DIP out of it, so here it is: > > http://wiki.dlang.org/DIP50 > > -- > /Jacob Carlborg >
Re: DIP 50 - AST macros
On 11/11/2013 02:49 AM, Timothee Cour wrote: People have shunned proposals to have @mixin functions because it wouldn't be obvious at call site that some statement is executed under a mixin (which could access all variables in scope etc). The same will happen here; ... macros are hygienic.
Re: DIP 50 - AST macros
On 2013-11-10 21:20:30 +, Jacob Carlborg said: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 Not a huge fan of the syntax, but I think it's a step in the right direction away from string mixins. R/ Shammah
Re: DIP 50 - AST macros
On 2013-11-11 03:04:34 +, Shammah Chancellor said: On 2013-11-10 21:20:30 +, Jacob Carlborg said: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 Not a huge fan of the syntax, but I think it's a step in the right direction away from string mixins. R/ Shammah I think this would gain a lot more traction if someone wrote an std.expressions which produced workable code from a AST. At that point, it's just for the D-frontend to parse code into said std.expressions. The .NET expressions framework might servce as some insperation. R/ Shammah
Re: DIP 50 - AST macros
On 2013-11-10 23:33, bearophile wrote: Jacob Carlborg: http://wiki.dlang.org/DIP50 I suggest to add some more use cases (possibly with their implementation). I could try and do that. The problem is I have not given much though on how the reflection API and the API used to build syntax trees should look like. If this get any serious interest and we start to think about the implementation I was hoping that someone we more experience could help out with designing the API. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 02:08, Rob T wrote: A scalable and elegant solution to the "inspection" problem may be a AST use case. Discussed here: http://forum.dlang.org/thread/qconpedgdkyeawmdz...@forum.dlang.org Please, feel free to add it as an example, if your interested. After all, it is a wiki. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 02:49, Timothee Cour wrote: People have shunned proposals to have @mixin functions because it wouldn't be obvious at call site that some statement is executed under a mixin (which could access all variables in scope etc). The same will happen here; I think it should be clear at call site that a macro is used. How about: macro!myAssert(1 + 2 == 4); instead of myAssert(1 + 2 == 4); I would really prefer if we not had to do that. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 02:40, Timothee Cour wrote:
<[ .. ]> syntax looks very foreign. How about m{...} (with q{...} as a
precedent)?
As the DIP says, that just one suggestion of the syntax. If we even need
a syntax. An alternative, without any new syntax is the "ast" macro:
http://wiki.dlang.org/DIP50#The_AST_Macro
This is similar to the "refiy" macro in Scala.
--
/Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 04:04, Shammah Chancellor wrote: Not a huge fan of the syntax, but I think it's a step in the right direction away from string mixins. To you mean the <[ ]> syntax or something else? See my reply to Timothee: http://forum.dlang.org/thread/l5otb1$1dhi$1...@digitalmars.com?page=2#post-l5q1vh:242c2k:241:40digitalmars.com -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote:
I've been thinking quite long of how AST macros could look like
in D. I've been posting my vision of AST macros here in the
newsgroup a couple of times already. I've now been asked to
create a DIP out of it, so here it is:
http://wiki.dlang.org/DIP50
One of our targets for AST macros should be the ability to
replicate roughly linq from c# / .net.
An example syntax for use with AST could be:
auto data = [5, 7, 9];
int[] data2;
query {
from value in data
where value >= 6
add to data2
}
Could be unwrapped to:
auto data = [5, 7, 9];
int[] data2;
foreach(value; data) {
if (value >= 6) data2 ~= value;
}
This isn't a thought out design but it should at least be a
target or a possibility.
Also c#'s get set should be rather similar as well.
Example:
getset {
public int id;
private bool exit;
}
Would translate to:
private int id;
private bool exit;
@property {
void id(int v) {this.id = v;}
void exit(bool v) { this.exit = v; }
int id() { return this.id; }
bool exit() { return this.exit; }
}
This would definitely open new possibilities up.
Disclaimer I don't like c# or .net but I am partial to these
features.
At current point I think the DIP does have the necessary features
to implement this. However it would be nice for safety to be able
to get all scope variables of where the macro was initiated from.
Being able to check for if a variable exists could provide much
needed compile safety and better error messages.
Re: DIP 50 - AST macros
On 2013-11-11 02:46, bearophile wrote: It's also useful to take a look at what F# is doing: http://tomasp.net/blog/2013/computation-zoo-padl/index.html I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 10.11.2013. 22:20, Jacob Carlborg wrote: > I've been thinking quite long of how AST macros could look like in D. > I've been posting my vision of AST macros here in the newsgroup a couple > of times already. I've now been asked to create a DIP out of it, so here > it is: > > http://wiki.dlang.org/DIP50 > Thumbs up!
Re: DIP 50 - AST macros
On 2013-11-11 09:08, luka8088 wrote: Thumbs up! Thanks. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 08:46, Rikki Cattermole wrote:
One of our targets for AST macros should be the ability to replicate
roughly linq from c# / .net.
An example syntax for use with AST could be:
auto data = [5, 7, 9];
int[] data2;
query {
from value in data
where value >= 6
add to data2
}
Could be unwrapped to:
auto data = [5, 7, 9];
int[] data2;
foreach(value; data) {
if (value >= 6) data2 ~= value;
}
This isn't a thought out design but it should at least be a target or a
possibility.
Absolutely. One of my favorite examples is the database query:
auto person = Person.where(e => e.name == "John");
Which translates to the following SQL:
select * from person where name = 'John'
Also c#'s get set should be rather similar as well.
Example:
getset {
public int id;
private bool exit;
}
Would translate to:
private int id;
private bool exit;
@property {
void id(int v) {this.id = v;}
void exit(bool v) { this.exit = v; }
int id() { return this.id; }
bool exit() { return this.exit; }
}
This would definitely open new possibilities up.
Disclaimer I don't like c# or .net but I am partial to these features.
That's quite similar one of the examples, I like to call it "property
shortcut":
http://wiki.dlang.org/DIP50#Attribute_macros
At current point I think the DIP does have the necessary features to
implement this. However it would be nice for safety to be able to get
all scope variables of where the macro was initiated from. Being able to
check for if a variable exists could provide much needed compile safety
and better error messages.
Why not? There's quite a lot that is not specified in this DIP. Mostly
because I haven't decided/figured out how it should work exactly. Of
course, any help is always appreciated.
--
/Jacob Carlborg
Re: DIP 50 - AST macros
On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote: Jacob Carlborg: http://wiki.dlang.org/DIP50 I suggest to add some more use cases (possibly with their implementation). Bye, bearophile I agree examples would help a lot. Trying to define what information actually exists within these types would also help a lot. In the first example, would Ast!(bool) be something like this? opBinary!"==" left = opBinary!"+" left = Literal type = int value = 1 right = Literal type = int value = 2 right = Literal type = int value = 4 Would there be helpers for matching part of the structure? The same applies to the other types used - what information should they have? As for examples, here's a couple of suggestions: * Expression to prefix notation * Expression to SQL * AutoImplement properties (like C#) * Number intervals, like "int i = int[10..20];" where only 10 to 20 are legal values * Discriminated union * Pattern matching
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 08:12:59 UTC, Jacob Carlborg wrote:
On 2013-11-11 08:46, Rikki Cattermole wrote:
One of our targets for AST macros should be the ability to
replicate
roughly linq from c# / .net.
An example syntax for use with AST could be:
auto data = [5, 7, 9];
int[] data2;
query {
from value in data
where value >= 6
add to data2
}
Could be unwrapped to:
auto data = [5, 7, 9];
int[] data2;
foreach(value; data) {
if (value >= 6) data2 ~= value;
}
This isn't a thought out design but it should at least be a
target or a
possibility.
Absolutely. One of my favorite examples is the database query:
auto person = Person.where(e => e.name == "John");
Which translates to the following SQL:
select * from person where name = 'John'
Also c#'s get set should be rather similar as well.
Example:
getset {
public int id;
private bool exit;
}
Would translate to:
private int id;
private bool exit;
@property {
void id(int v) {this.id = v;}
void exit(bool v) { this.exit = v; }
int id() { return this.id; }
bool exit() { return this.exit; }
}
This would definitely open new possibilities up.
Disclaimer I don't like c# or .net but I am partial to these
features.
That's quite similar one of the examples, I like to call it
"property shortcut":
http://wiki.dlang.org/DIP50#Attribute_macros
At current point I think the DIP does have the necessary
features to
implement this. However it would be nice for safety to be able
to get
all scope variables of where the macro was initiated from.
Being able to
check for if a variable exists could provide much needed
compile safety
and better error messages.
Why not? There's quite a lot that is not specified in this DIP.
Mostly because I haven't decided/figured out how it should work
exactly. Of course, any help is always appreciated.
Theres a few other things I think would need to be cleared up.
For example take this code:
shader {
program
vertex {
#version 150
void main() {
}
}
fragment {
#version 150
void main() {
}
}
}
At this point the vertex / fragment segments would be treated as
strings instead of calling their macros. So perhaps a rule to
identify macros inside macros calls? It would save a lot of time
for parsing reasons.
Also can you alias a macro?
alias fragment = tostring;
That would make things a lot simpler creating nice structures of
them.
Perhaps a condition on a macro like the if's we got for templates
would be useful in the sense of being able to say:
if (lexer.compareLine(0, "[a-zA-Z]{1}[a-zA-Z_0-9]*") &&
lexer.isSymbol(1, SymbolTypes.Macro, vertex))
This would require us to develop a lexer library. But if done
right I don't see why it wouldn't be usable for more than just D
macro checks. Preferably also for e.g. c/c++ wink wink for when
D's front end is in D.
Re: DIP 50 - AST macros
On 2013-11-11 09:28, simendsjo wrote: I agree examples would help a lot. Trying to define what information actually exists within these types would also help a lot. I can tell you right now that I haven't figured out everything, far from everything. In the first example, would Ast!(bool) be something like this? opBinary!"==" left = opBinary!"+" left = Literal type = int value = 1 right = Literal type = int value = 2 right = Literal type = int value = 4 Would there be helpers for matching part of the structure? I don't know. As I said above, I haven't figured out everything. I don't see any support for this in the language. But of course, there can be library functions that help with this. The same applies to the other types used - what information should they have? Suggestions are welcome. Also, see my reply to bearophile http://forum.dlang.org/thread/l5otb1$1dhi$1...@digitalmars.com#post-l5q1p5:242bs4:241:40digitalmars.com As for examples, here's a couple of suggestions: * Expression to prefix notation * Expression to SQL * AutoImplement properties (like C#) Would that be something like the attribute macro example? http://wiki.dlang.org/DIP50#Attribute_macros * Number intervals, like "int i = int[10..20];" where only 10 to 20 are legal values * Discriminated union * Pattern matching Pattern matching usually requires new syntax, which this proposal does not support. -- /Jacob Carlborg
Re: DIP 50 - AST macros
Jacob Carlborg: One of my favorite examples is the database query: auto person = Person.where(e => e.name == "John"); Which translates to the following SQL: select * from person where name = 'John' Can't you do the same thing with functions similar (same API but different semantics) to std.algorithm ones that generate expression templates? auto person = persons.filter!(e => e.name == "John"); simendsjo: * Number intervals, like "int i = int[10..20];" where only 10 to 20 are legal values What's wrong with this syntax that doesn't reqiire macros? It's more uniform with the rest of the language: Ranged!(int, 10, 20) i; Bye, bearophile
Re: DIP 50 - AST macros
One of our targets for AST macros should be the ability to
replicate roughly linq from c# / .net.
An example syntax for use with AST could be:
auto data = [5, 7, 9];
int[] data2;
query {
from value in data
where value >= 6
add to data2
}
Could be unwrapped to:
auto data = [5, 7, 9];
int[] data2;
foreach(value; data) {
if (value >= 6) data2 ~= value;
}
could you add the example to the DIP wiki page
Re: DIP 50 - AST macros
Am 11.11.2013 09:28, schrieb simendsjo: On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote: Jacob Carlborg: http://wiki.dlang.org/DIP50 I suggest to add some more use cases (possibly with their implementation). Bye, bearophile I agree examples would help a lot. Trying to define what information actually exists within these types would also help a lot. In the first example, would Ast!(bool) be something like this? opBinary!"==" left = opBinary!"+" left = Literal type = int value = 1 right = Literal type = int value = 2 right = Literal type = int value = 4 Would there be helpers for matching part of the structure? The same applies to the other types used - what information should they have? As for examples, here's a couple of suggestions: * Expression to prefix notation * Expression to SQL * AutoImplement properties (like C#) * Number intervals, like "int i = int[10..20];" where only 10 to 20 are legal values * Discriminated union * Pattern matching can you add the example to the DIP?
Re: DIP 50 - AST macros
On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 I like it, Jacob. With or without <[ ]> it is a good proposal, and I see several places where I would use it. +1
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 09:02:01 UTC, bearophile wrote: Jacob Carlborg: One of my favorite examples is the database query: auto person = Person.where(e => e.name == "John"); Which translates to the following SQL: select * from person where name = 'John' Can't you do the same thing with functions similar (same API but different semantics) to std.algorithm ones that generate expression templates? auto person = persons.filter!(e => e.name == "John"); The problem here is that a library need to know that it has to create SELECT * FROM persons WHERE name = 'John'; and not SELECT * FROM persons; and filter it locally. So it needs a way to inspect the body of the delegate and extract "name" "==" and "John". simendsjo: * Number intervals, like "int i = int[10..20];" where only 10 to 20 are legal values What's wrong with this syntax that doesn't reqiire macros? It's more uniform with the rest of the language: Ranged!(int, 10, 20) i; Bye, bearophile Nothing wrong with it, I was just trying to come up with some examples. I don't say they're necessarily good :)
Re: DIP 50 - AST macros
On 2013-11-11 09:31, Rikki Cattermole wrote:
Theres a few other things I think would need to be cleared up.
For example take this code:
shader {
program
vertex {
#version 150
void main() {
}
}
fragment {
#version 150
void main() {
}
}
}
At this point the vertex / fragment segments would be treated as strings
instead of calling their macros. So perhaps a rule to identify macros
inside macros calls? It would save a lot of time for parsing reasons.
They way I see that is the AST of the whole block would be passed to
"shader". I have though a bit about macros inside macros, but I haven't
come to a conclusion. The easiest way from a design point of view seems
to be that "shader" to basically return the whole AST it receives plus
any addition it needs to do. The the "vertex" and "fragment" macros are
expanded recursively.
Also can you alias a macro?
alias fragment = tostring;
That would make things a lot simpler creating nice structures of them.
I don't see why not.
Perhaps a condition on a macro like the if's we got for templates would
be useful in the sense of being able to say:
if (lexer.compareLine(0, "[a-zA-Z]{1}[a-zA-Z_0-9]*") &&
lexer.isSymbol(1, SymbolTypes.Macro, vertex))
I don't think I understand how this should be used.
This would require us to develop a lexer library. But if done right I
don't see why it wouldn't be usable for more than just D macro checks.
Preferably also for e.g. c/c++ wink wink for when D's front end is in D.
--
/Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 10:02, bearophile wrote: Can't you do the same thing with functions similar (same API but different semantics) to std.algorithm ones that generate expression templates? auto person = persons.filter!(e => e.name == "John"); I have been thinking about that, a solution where "e" is a proxy that implements opDispatch and which returns a new proxy which overloads "==". The problem with this is that you cannot separately overload the equal and comparison operators. In D the all the comparisons operators are implement with the single overload, opCmp. You cannot know from inside of opCmp if it's the "<" or the ">" operator that is being called. Same problem with opEquals. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 10:07, Dejan Lekic wrote: I like it, Jacob. With or without <[ ]> it is a good proposal, and I see several places where I would use it. +1 Thank you. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 10:23, simendsjo wrote: On Monday, 11 November 2013 at 09:02:01 UTC, bearophile wrote: auto person = persons.filter!(e => e.name == "John"); The problem here is that a library need to know that it has to create SELECT * FROM persons WHERE name = 'John'; and not SELECT * FROM persons; and filter it locally. So it needs a way to inspect the body of the delegate and extract "name" "==" and "John". This particular example would work. "e" would be a proxy which implements opDispatch which returns another proxy which overloads opEquals. The problem is that != and all the comparison operators won't work. See my reply to bearophile: http://forum.dlang.org/thread/l5otb1$1dhi$1...@digitalmars.com?page=3#post-l5q892:242kpf:241:40digitalmars.com -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 09:24:31 UTC, Jacob Carlborg wrote:
On 2013-11-11 09:31, Rikki Cattermole wrote:
Theres a few other things I think would need to be cleared up.
For example take this code:
shader {
program
vertex {
#version 150
void main() {
}
}
fragment {
#version 150
void main() {
}
}
}
At this point the vertex / fragment segments would be treated
as strings
instead of calling their macros. So perhaps a rule to identify
macros
inside macros calls? It would save a lot of time for parsing
reasons.
They way I see that is the AST of the whole block would be
passed to "shader". I have though a bit about macros inside
macros, but I haven't come to a conclusion. The easiest way
from a design point of view seems to be that "shader" to
basically return the whole AST it receives plus any addition it
needs to do. The the "vertex" and "fragment" macros are
expanded recursively.
Also can you alias a macro?
alias fragment = tostring;
That would make things a lot simpler creating nice structures
of them.
I don't see why not.
Perhaps a condition on a macro like the if's we got for
templates would
be useful in the sense of being able to say:
if (lexer.compareLine(0, "[a-zA-Z]{1}[a-zA-Z_0-9]*") &&
lexer.isSymbol(1, SymbolTypes.Macro, vertex))
I don't think I understand how this should be used.
This would require us to develop a lexer library. But if done
right I
don't see why it wouldn't be usable for more than just D macro
checks.
Preferably also for e.g. c/c++ wink wink for when D's front
end is in D.
An example of this might be:
macro foo (Context context, Ast!(string) str)
if (lexer.isSymbol(0, SymbolTypes.Class))
{
return str;
}
class Bar {
int i;
}
foo {
Bar(7);
}
The above code would succeed however the below code will give a
compiler error stating that the given macro is not found.
struct Haz {
int i;
}
foo {
Haz(9);
}
The above code is based on the assumption of a D lexer in phobos.
This is simpler set of code which won't require a lexer.
macro foo (Context context, Ast!(string) str)
if (str == "working")
{
return "";
}
foo {fails}
foo {working}
In these cases its a simple string test to determine if the text
given is specified value.
In the original example I gave, I was using regex to make the
point of validation for a given line passed to the macro.
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 09:01:31 UTC, dennis luehring
wrote:
One of our targets for AST macros should be the ability to
replicate roughly linq from c# / .net.
An example syntax for use with AST could be:
auto data = [5, 7, 9];
int[] data2;
query {
from value in data
where value >= 6
add to data2
}
Could be unwrapped to:
auto data = [5, 7, 9];
int[] data2;
foreach(value; data) {
if (value >= 6) data2 ~= value;
}
could you add the example to the DIP wiki page
Adding use case section with Linq example.
Please remove / modify as required as is first time editing wiki.
Re: DIP 50 - AST macros
On 2013-11-11 10:38, Rikki Cattermole wrote:
An example of this might be:
macro foo (Context context, Ast!(string) str)
if (lexer.isSymbol(0, SymbolTypes.Class))
{
return str;
}
class Bar {
int i;
}
foo {
Bar(7);
}
The above code would succeed however the below code will give a compiler
error stating that the given macro is not found.
struct Haz {
int i;
}
foo {
Haz(9);
}
The above code is based on the assumption of a D lexer in phobos.
This is simpler set of code which won't require a lexer.
macro foo (Context context, Ast!(string) str)
if (str == "working")
{
return "";
}
foo {fails}
foo {working}
In these cases its a simple string test to determine if the text given
is specified value.
In the original example I gave, I was using regex to make the point of
validation for a given line passed to the macro.
My idea of handling errors in macros is more something like triggering a
real compile error. That could be done via the context parameter,
something like:
macro foo (Context context, Ast!(string) str)
{
if (str.eval() == "bar")
context.error("Illegal value 'bar'"); // this will trigger the
compile error
}
--
/Jacob Carlborg
Re: DIP 50 - AST macros
I would really like to use AST macros for the following use case:
myAssert( x < y);
//will print, on failure, a message, along with all values appearing inside
macro myAssert:
xwrote:
> On Monday, 11 November 2013 at 09:01:31 UTC, dennis luehring wrote:
>
>> One of our targets for AST macros should be the ability to
>>> replicate roughly linq from c# / .net.
>>>
>>> An example syntax for use with AST could be:
>>>
>>> auto data = [5, 7, 9];
>>> int[] data2;
>>> query {
>>> from value in data
>>> where value >= 6
>>> add to data2
>>> }
>>>
>>> Could be unwrapped to:
>>>
>>> auto data = [5, 7, 9];
>>> int[] data2;
>>> foreach(value; data) {
>>> if (value >= 6) data2 ~= value;
>>> }
>>>
>>
>> could you add the example to the DIP wiki page
>>
>
> Adding use case section with Linq example.
> Please remove / modify as required as is first time editing wiki.
>
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 10:04:31 UTC, Jacob Carlborg wrote:
On 2013-11-11 10:38, Rikki Cattermole wrote:
An example of this might be:
macro foo (Context context, Ast!(string) str)
if (lexer.isSymbol(0, SymbolTypes.Class))
{
return str;
}
class Bar {
int i;
}
foo {
Bar(7);
}
The above code would succeed however the below code will give
a compiler
error stating that the given macro is not found.
struct Haz {
int i;
}
foo {
Haz(9);
}
The above code is based on the assumption of a D lexer in
phobos.
This is simpler set of code which won't require a lexer.
macro foo (Context context, Ast!(string) str)
if (str == "working")
{
return "";
}
foo {fails}
foo {working}
In these cases its a simple string test to determine if the
text given
is specified value.
In the original example I gave, I was using regex to make the
point of
validation for a given line passed to the macro.
My idea of handling errors in macros is more something like
triggering a real compile error. That could be done via the
context parameter, something like:
macro foo (Context context, Ast!(string) str)
{
if (str.eval() == "bar")
context.error("Illegal value 'bar'"); // this will
trigger the compile error
}
For errors I was thinking a new pragma.
pragma(error, "You did x wrong");
This would fire an error and display the text given. Returning
error code as expected. It is after all in the same league as msg.
It would also remove our current static asserts in our code.
So other words it could be replicated by:
pragma(msg, "You did x wrong");
static assert(0);
Just without the whole failed assert message.
Although what I was showing was the idea of overloading of macros
like we have in templates.
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 10:10:33 UTC, Timothee Cour wrote:
I would really like to use AST macros for the following use
case:
myAssert( x < y);
//will print, on failure, a message, along with all values
appearing inside
macro myAssert:
x//likewise, but nesting down to all individual variables
appearing inside
the macro:
x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=...
This would advantageously replace the plethora of unittest
helpers found in
other languages, eg: CHECK_EQ, CHECK_LEQ, etc.
Invaluable for debugging or informative unittests and logs.
On Mon, Nov 11, 2013 at 1:52 AM, Rikki Cattermole
wrote:
On Monday, 11 November 2013 at 09:01:31 UTC, dennis luehring
wrote:
One of our targets for AST macros should be the ability to
replicate roughly linq from c# / .net.
An example syntax for use with AST could be:
auto data = [5, 7, 9];
int[] data2;
query {
from value in data
where value >= 6
add to data2
}
Could be unwrapped to:
auto data = [5, 7, 9];
int[] data2;
foreach(value; data) {
if (value >= 6) data2 ~= value;
}
could you add the example to the DIP wiki page
Adding use case section with Linq example.
Please remove / modify as required as is first time editing
wiki.
Perhaps an alternative would be:
asserts {
x < y
z == z
}
Where the macro would do something like this once converted:
if (!(x < y)) writeln("x < y failed with x=", x, " y= ", y);
if (!(z == z)) writeln("z == z failed with z=", z);
This would also have the benefit of allowing for multiple assert
statements for one block.
Re: DIP 50 - AST macros
On 2013-11-11 11:11, Rikki Cattermole wrote: For errors I was thinking a new pragma. pragma(error, "You did x wrong"); This would fire an error and display the text given. Returning error code as expected. It is after all in the same league as msg. It would also remove our current static asserts in our code. So other words it could be replicated by: pragma(msg, "You did x wrong"); static assert(0); Just without the whole failed assert message. I like the context parameter better. I'm imagine the context parameter containing a lot of information and functionality to interact with the compiler. A pragma seems not be general enough. Although what I was showing was the idea of overloading of macros like we have in templates. Ok, I see. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 11:10, Timothee Cour wrote: I would really like to use AST macros for the following use case: myAssert( x < y); //will print, on failure, a message, along with all values appearing inside macro myAssert: x Agree. That's the first example in the DIP. Although a very simplified version. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 10:19:49 UTC, Jacob Carlborg wrote: On 2013-11-11 11:11, Rikki Cattermole wrote: For errors I was thinking a new pragma. pragma(error, "You did x wrong"); This would fire an error and display the text given. Returning error code as expected. It is after all in the same league as msg. It would also remove our current static asserts in our code. So other words it could be replicated by: pragma(msg, "You did x wrong"); static assert(0); Just without the whole failed assert message. I like the context parameter better. I'm imagine the context parameter containing a lot of information and functionality to interact with the compiler. A pragma seems not be general enough. I can understand wanting to have a high level version of it in context. I'm just considering a high reuse language feature that could back it. Really it should have been added a while ago given the static assert trick is used a bit. I do actually like the idea of having the ability to call it on context with a bunch of extra information given to it for you. e.g. line number of initiation ext.
Re: DIP 50 - AST macros
On Mon, Nov 11, 2013 at 2:21 AM, Jacob Carlborg wrote: > On 2013-11-11 11:10, Timothee Cour wrote: > >> I would really like to use AST macros for the following use case: >> >> myAssert( x < y); >> //will print, on failure, a message, along with all values appearing >> inside macro myAssert: >> x> >> myAssert( fun(x,y)==z1+z2) >> //likewise, but nesting down to all individual variables appearing >> inside the macro: >> x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=... >> >> This would advantageously replace the plethora of unittest helpers found >> in other languages, eg: CHECK_EQ, CHECK_LEQ, etc. >> Invaluable for debugging or informative unittests and logs. >> > > Agree. That's the first example in the DIP. Although a very simplified > version. yes, I think your initial example becomes more interesting with this, as in its current form it can already be done in current D. This would make error messages self-documenting: myAssert(!file.exists ); // "!file. exists" failed: dump of AST: file: string ="foobar.d" |_file.exists: bool = false |_!file. exists: bool = false In many cases, this would be so much more useful than an out of date / incomplete string error message, esp w a lot of variables involved.
Re: DIP 50 - AST macros
On 2013-11-11 11:28, Rikki Cattermole wrote: I can understand wanting to have a high level version of it in context. I'm just considering a high reuse language feature that could back it. Really it should have been added a while ago given the static assert trick is used a bit. I do actually like the idea of having the ability to call it on context with a bunch of extra information given to it for you. e.g. line number of initiation ext. I think we want to have as much as possible of this collected in the same place. I don't think pragmas are a good fit for this. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 10:47:37 UTC, Jacob Carlborg wrote: On 2013-11-11 11:28, Rikki Cattermole wrote: I can understand wanting to have a high level version of it in context. I'm just considering a high reuse language feature that could back it. Really it should have been added a while ago given the static assert trick is used a bit. I do actually like the idea of having the ability to call it on context with a bunch of extra information given to it for you. e.g. line number of initiation ext. I think we want to have as much as possible of this collected in the same place. I don't think pragmas are a good fit for this. Yes I do agree with that. I'm just more worried about what backs it then pleasantries that we can give the developer. Also if you're not looking towards pragma's to hook into stopping compilation what are your ideas and thoughts around this issue? Because at this point I only know of static assert to do it.
Re: DIP 50 - AST macros
On 2013-11-11 11:39, Timothee Cour wrote: yes, I think your initial example becomes more interesting with this, as in its current form it can already be done in current D. The current example prints out the exact expression what was passed to the assert macro. Not the result of evaluating the expression. I don't see how this can currently be done in D This would make error messages self-documenting: myAssert(!file.exists ); // "!file. exists" failed: dump of AST: file: string ="foobar.d" |_file.exists: bool = false |_!file. exists: bool = false In many cases, this would be so much more useful than an out of date / incomplete string error message, esp w a lot of variables involved. I agree. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 10:55:23 UTC, Jacob Carlborg wrote: The current example prints out the exact expression what was passed to the assert macro. Not the result of evaluating the expression. I don't see how this can currently be done in D I made a suggestion regarding a macro able to get scoped variables from initiation point with Linq. And why that would be a good thing. This is another use case for it. Perhaps add it as a use case for having that feature on the DIP?
Re: DIP 50 - AST macros
On 2013-11-11 11:51, Rikki Cattermole wrote: Yes I do agree with that. I'm just more worried about what backs it then pleasantries that we can give the developer. Also if you're not looking towards pragma's to hook into stopping compilation what are your ideas and thoughts around this issue? Because at this point I only know of static assert to do it. A function call on the context parameter as I've already showed. This would just call "error" which is already implemented in the compiler. Although I don't know how to actually do the connection between the compiler internals the user code. I think this is one of the big challenges with this DIP. For this function I guess it could be done like an intrinsic (is that the correct word?). -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 10:59:28 UTC, Jacob Carlborg wrote: A function call on the context parameter as I've already showed. This would just call "error" which is already implemented in the compiler. Although I don't know how to actually do the connection between the compiler internals the user code. I think this is one of the big challenges with this DIP. For this function I guess it could be done like an intrinsic (is that the correct word?). Ok pragmas essentially call functions like error in the compiler [0]. What I am thinking error will do is instead of outputting like msg it'll call error[1]. This is exactly what you want. [0] https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2862 [1] https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2957
Re: DIP 50 - AST macros
On 2013-11-11 11:58, Rikki Cattermole wrote: I made a suggestion regarding a macro able to get scoped variables from initiation point with Linq. And why that would be a good thing. This is another use case for it. Perhaps add it as a use case for having that feature on the DIP? I already have an example with the assert (not among the use cases but at the top). Or were you referring to something else? -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Mon, Nov 11, 2013 at 2:55 AM, Jacob Carlborg wrote: > On 2013-11-11 11:39, Timothee Cour wrote: > > yes, I think your initial example becomes more interesting with this, as >> in its current form it can already be done in current D. >> > > The current example prints out the exact expression what was passed to the > assert macro. Not the result of evaluating the expression. I don't see how > this can currently be done in D I'm using a modified assert that uses the ugly import(file)[line] trick, so that can be done (but is really ugly and has impact either runtime or compile time). But we're essentially in agreement. > > > This would make error messages self-documenting: >> >> myAssert(!file.exists ); >> // "!file. exists" failed: dump of AST: >> file: string ="foobar.d" >> |_file.exists: bool = false >>|_!file. exists: bool = false >> >> In many cases, this would be so much more useful than an out of date / >> incomplete string error message, esp w a lot of variables involved. >> > > I agree. > > -- > /Jacob Carlborg >
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 11:01:41 UTC, Jacob Carlborg wrote:
I already have an example with the assert (not among the use
cases but at the top). Or were you referring to something else?
I was referring to the use case as to why we should have the
ability to get scoped variables from the initiation point.
Example:
void func(int i) {
bool b;
macr {}
}
macro foo (Context context, Ast!(string) str)
{
writeln(context.scopeVariables!int("i"));
writeln(context.scopeVariables!bool("b"));
return "";
}
Re: DIP 50 - AST macros
11-Nov-2013 01:20, Jacob Carlborg пишет: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 I have to say I like it. I find it nice that it would allow one to finally ensure that inlining is always done by making critical primitives macros instead of functions. This is vastly superior compared to the _only_ current option of generating unrolled code with string mixins. I'd call attribute macros - declaration macros in line with statement macros, then describe how they look - a lot like attributes. Things to note though: - Can't implement scope(exit/success/failure) because that would need to capture the outer scope to pack it into try/catch/finally. - Can't implement foreach if only because of the syntax of the latter. - There is no definition of Ast!(T) construct - I take it's a struct or a class of sorts and the whole thing works with the CTFE engine. Then it also needs specification of Statements, Declarations. - It seems like I can use AST literal in R-T code as well? That would be useful but only if some library actually accepted these AST nodes. -- Dmitry Olshansky
Re: DIP 50 - AST macros
Am Sun, 10 Nov 2013 22:20:30 +0100 schrieb Jacob Carlborg : > I've been thinking quite long of how AST macros could look like in D. > I've been posting my vision of AST macros here in the newsgroup a couple > of times already. I've now been asked to create a DIP out of it, so here > it is: > > http://wiki.dlang.org/DIP50 Wow, some topics really explode recently. Is the D user base growing or are AST macros drawing so much attention? -- Marco
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 12:13:54 UTC, Marco Leise wrote: Wow, some topics really explode recently. Is the D user base growing or are AST macros drawing so much attention? I'd like to say both. AST macros are one of those few things that I wouldn't mind seeing breaking changes for and I'm in the camp of "avoid breaking changes at all cost". AST macros are to code what templates are to data. Despite it being possible to go without templates, I'm not at all tempted to switch to Go.
Re: DIP 50 - AST macros
On 2013-11-11 12:04, Rikki Cattermole wrote: Ok pragmas essentially call functions like error in the compiler [0]. What I am thinking error will do is instead of outputting like msg it'll call error[1]. This is exactly what you want. [0] https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2862 [1] https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2957 Yes, I still don't understand why you would want it as a pragma. Be usable outside of macros? -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 12:08, Rikki Cattermole wrote:
I was referring to the use case as to why we should have the ability to
get scoped variables from the initiation point.
Example:
void func(int i) {
bool b;
macr {}
}
macro foo (Context context, Ast!(string) str)
{
writeln(context.scopeVariables!int("i"));
writeln(context.scopeVariables!bool("b"));
return "";
}
Ok, I see. Please add additional examples to the DIP if you think it's
missing.
--
/Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 12:06, Timothee Cour wrote: I'm using a modified assert that uses the ugly import(file)[line] trick, so that can be done (but is really ugly and has impact either runtime or compile time). But we're essentially in agreement. Right, that ugly trick :) -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 12:30:07 UTC, Jacob Carlborg wrote: Yes, I still don't understand why you would want it as a pragma. Be usable outside of macros? Yes outside of macros would be useful. For example code like this would become redundant: pragma(msg, "Support for x is not implemented on platform y"); static assert(0); Becoming: pragma(error, "Support for x is not implemented on platform y"); Also pragma's core responsibility is to cause the compiler to do something. In this case to say we hit an error during compilation please tell the user/dev and die. It is a hook to the compilers workings. Currently working on getting this implemented. Nearly done with it. Just got some extra spaces that shouldn't be in output.
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 01:49:45 UTC, Timothee Cour wrote: People have shunned proposals to have @mixin functions because it wouldn't be obvious at call site that some statement is executed under a mixin (which could access all variables in scope etc). The same will happen here; I think it should be clear at call site that a macro is used. How about: macro!myAssert(1 + 2 == 4); instead of myAssert(1 + 2 == 4); If macros are supposed to access outer scope, I agree, this is a necessary restriction.
Re: DIP 50 - AST macros
On 2013-11-11 12:34, Dmitry Olshansky wrote:
I have to say I like it.
Cool :)
I find it nice that it would allow one to finally ensure that inlining
is always done by making critical primitives macros instead of
functions. This is vastly superior compared to the _only_ current option
of generating unrolled code with string mixins.
I'd call attribute macros - declaration macros in line with statement
macros, then describe how they look - a lot like attributes.
Ok, I can change that.
Things to note though:
- Can't implement scope(exit/success/failure) because that would need
to capture the outer scope to pack it into try/catch/finally.
Right. scope(exit) could perhaps be implement with a custom struct and
calling a block in the destructor.
- Can't implement foreach if only because of the syntax of the latter.
I would think something like this:
macro foreach (T) (Context context, Ast!(Symbol) var, Ast!(T)
collection, Statement block)
{
if (isRange(collection))
{
return <[
for (auto __r = $collection; !__r.empty; __r.popFront())
{
auto $var = __r.front;
block;
}
]>;
}
else if (hasOpApply(collection))
{
...
}
else ...
}
Perhaps not all of the existing syntax can fit in a macro.
- There is no definition of Ast!(T) construct - I take it's a struct
or a class of sorts and the whole thing works with the CTFE engine. Then
it also needs specification of Statements, Declarations.
Yes, something like that. I don't know exactly how the Ast type need to
look like. See one of my replies to bearophile:
http://forum.dlang.org/thread/l5otb1$1dhi$1...@digitalmars.com
- It seems like I can use AST literal in R-T code as well? That would
be useful but only if some library actually accepted these AST nodes.
Actually, I haven't thought about this. I think my original idea was
that <[ ]> should only work inside macros. But that it would be possible
to pass Ast nodes to other functions.
--
/Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 13:13, Marco Leise wrote: Wow, some topics really explode recently. Is the D user base growing or are AST macros drawing so much attention? AST macros are drawing so much attension. For some, it's like the holy grail. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 13:36, Rikki Cattermole wrote: Yes outside of macros would be useful. For example code like this would become redundant: pragma(msg, "Support for x is not implemented on platform y"); static assert(0); Becoming: pragma(error, "Support for x is not implemented on platform y"); Also pragma's core responsibility is to cause the compiler to do something. In this case to say we hit an error during compilation please tell the user/dev and die. It is a hook to the compilers workings. Currently working on getting this implemented. Nearly done with it. Just got some extra spaces that shouldn't be in output. I just don't want to add a bunch of pragmas. I don't know what's the best solution. -- /Jacob Carlborg
Re: DIP 50 - AST macros
Am 11.11.2013 13:36, schrieb Rikki Cattermole: On Monday, 11 November 2013 at 12:30:07 UTC, Jacob Carlborg wrote: Yes, I still don't understand why you would want it as a pragma. Be usable outside of macros? Yes outside of macros would be useful. For example code like this would become redundant: pragma(msg, "Support for x is not implemented on platform y"); static assert(0); Becoming: pragma(error, "Support for x is not implemented on platform y"); Also pragma's core responsibility is to cause the compiler to do something. In this case to say we hit an error during compilation please tell the user/dev and die. It is a hook to the compilers workings. Currently working on getting this implemented. Nearly done with it. Just got some extra spaces that shouldn't be in output. but in macros the context information is much more interesting then anything else - so #pragma(error,...) won't fully fit the needs to context error returning - it could be much more then just a message example: to help the compiler forming better error messages - or maybe recover from deep macro expansion etc...) what you want is just #pragma(error,...) -> break compiliation now Jacob is talking about the feedback for the compiler thing...
Re: DIP 50 - AST macros
On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 I don't have time to investigate this in details but I'd like to mention that this is one of few possible new features where ROI is really worth it. It removes lot of burden from core language and allows to replace some very arcane template hacks with cleaner code. I'd say it is a bit too early to go for it (really, we need at least to figure out how to make releases without issues first :)) but it is a good future goal even in context of general feature freezing.
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 13:16:23 UTC, Jacob Carlborg wrote:
I just don't want to add a bunch of pragmas. I don't know
what's the best solution.
I am only suggesting one :)
We only need one that'll output any text we want it to and make
the compiler consider it an error.
On the macro side of things we can build a string which then can
be given to the pragma.
All of the wrapping giving the goodness would go into the context.
Current what I have enables this:
void main() {
U u;
}
alias T!("hi", "bye") U;
struct T(string a, string b) {
pragma(error, a ~ "\n" ~ b);
}
test.d(8): Error: hi
bye
test.d(5): Error: template instance test.T!("hi", "bye") error
instantiating
Not really in pull state but thats the best I can do. Really that
final error should be omitted.
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 13:22:34 UTC, dennis luehring wrote: example: to help the compiler forming better error messages - or maybe recover from deep macro expansion etc...) what you want is just #pragma(error,...) -> break compiliation now Jacob is talking about the feedback for the compiler thing... Jacob mentioned originally he wanted to create a compiler error. Or at least thats is how I took it. Hence pragma(error, "text"); We can already message out at both compile and runtime its simply does the compiler recognise it as an error part.
Re: DIP 50 - AST macros
On 2013-11-11 14:32, Rikki Cattermole wrote:
I am only suggesting one :)
What about warnings? And no, the current pragma(msg) isn't the same.
We only need one that'll output any text we want it to and make the
compiler consider it an error.
On the macro side of things we can build a string which then can be
given to the pragma.
All of the wrapping giving the goodness would go into the context.
Current what I have enables this:
void main() {
U u;
}
alias T!("hi", "bye") U;
struct T(string a, string b) {
pragma(error, a ~ "\n" ~ b);
}
test.d(8): Error: hi
bye
test.d(5): Error: template instance test.T!("hi", "bye") error
instantiating
Not really in pull state but thats the best I can do. Really that final
error should be omitted.
I wouldn't say no to this if macros are completely off the table.
Perhaps it's a good addition regardless.
--
/Jacob Carlborg
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 13:36:14 UTC, Jacob Carlborg wrote: What about warnings? And no, the current pragma(msg) isn't the same. I'm not doing one for warnings. See how the error one goes. The warning one won't stop compilation and pragma msg can be used to fake it. I wouldn't say no to this if macros are completely off the table. Perhaps it's a good addition regardless. Personally I think it was already needed even without macros. Just this has started me off on actually looking into it. We'll see what the compiler devs think though.
Re: DIP 50 - AST macros
On 2013-11-11 14:36, Rikki Cattermole wrote: Jacob mentioned originally he wanted to create a compiler error. Or at least thats is how I took it. Hence pragma(error, "text"); We can already message out at both compile and runtime its simply does the compiler recognise it as an error part. "error" would be just a single function that is available in the context parameter. There would be many others. From the DIP: * The arguments used when the compiler was invoked * Functions for emitting messages of various verbosity level, like error, warning and info * Functions for querying various types of settings/options, like which versions are defined, is "debug" or "release" defined and so on * In general providing as much as possible of what the compiler knows about the compile run * The context should have an associative array with references to all scoped variables at initiation point. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 14:40, Rikki Cattermole wrote: I'm not doing one for warnings. See how the error one goes. The warning one won't stop compilation and pragma msg can be used to fake it. Yes, but I want warnings to be available in the context parameter as well. That's what I've been talking about all along. It's not just errors, it's all of the stuff that can be added. Warnings can be turned off, pragma(msg) cannot (at least not as far as I know). Personally I think it was already needed even without macros. Just this has started me off on actually looking into it. We'll see what the compiler devs think though. I see. I'm not saying that we don't need it. -- /Jacob Carlborg
Re: DIP 50 - AST macros
Jacob Carlborg: AST macros are drawing so much attension. For some, it's like the holy grail. That's a bad approach to technology design. You first need to create a good list of things that you want to do, then you have to try to do them without new features, and then if something important is not possible without the new feature (or it becomes too much ugly, too much long, too much bug-prone, etc), you have to see if the complexity of introducing the new feature is paid back. Macros are a significant feature, so they have a significant cost in code complexity, language dis-uniformity, ecc. Bye, bearophile
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 14:13:31 UTC, Jacob Carlborg wrote: On 2013-11-11 14:36, Rikki Cattermole wrote: Jacob mentioned originally he wanted to create a compiler error. Or at least thats is how I took it. Hence pragma(error, "text"); We can already message out at both compile and runtime its simply does the compiler recognise it as an error part. "error" would be just a single function that is available in the context parameter. There would be many others. From the DIP: * The arguments used when the compiler was invoked * Functions for emitting messages of various verbosity level, like error, warning and info * Functions for querying various types of settings/options, like which versions are defined, is "debug" or "release" defined and so on * In general providing as much as possible of what the compiler knows about the compile run * The context should have an associative array with references to all scoped variables at initiation point. Is there anything specific for info that I should look at? or does pragma msg do everything for you?
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 14:10:32 UTC, Jacob Carlborg wrote: On 2013-11-11 14:40, Rikki Cattermole wrote: I'm not doing one for warnings. See how the error one goes. The warning one won't stop compilation and pragma msg can be used to fake it. Yes, but I want warnings to be available in the context parameter as well. That's what I've been talking about all along. It's not just errors, it's all of the stuff that can be added. Warnings can be turned off, pragma(msg) cannot (at least not as far as I know). Ah ok, I'll see what I can drum up in that regard then. I guess I was wrong about being able to fake it with pragma msg.
Re: DIP 50 - AST macros
On 2013-11-11 16:24, Rikki Cattermole wrote: Is there anything specific for info that I should look at? or does pragma msg do everything for you? Sorry, I don't understand what you're meaning. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 11/11/2013 01:36 PM, Rikki Cattermole wrote: Yes outside of macros would be useful. For example code like this would become redundant: pragma(msg, "Support for x is not implemented on platform y"); static assert(0); Becoming: pragma(error, "Support for x is not implemented on platform y"); static assert(0, "Support for x is not implemented on platform y");
Re: DIP 50 - AST macros
On 11/11/2013 01:55 PM, Dicebot wrote: On Monday, 11 November 2013 at 01:49:45 UTC, Timothee Cour wrote: People have shunned proposals to have @mixin functions because it wouldn't be obvious at call site that some statement is executed under a mixin (which could access all variables in scope etc). The same will happen here; I think it should be clear at call site that a macro is used. How about: macro!myAssert(1 + 2 == 4); instead of myAssert(1 + 2 == 4); If macros are supposed to access outer scope, I agree, this is a necessary restriction. I don't agree. (The argument against implicit mixin features was a complete lack of hygiene, not the mere possibility of accessing the caller scope.)
Re: DIP 50 - AST macros
On 11/11/2013 03:13 PM, Jacob Carlborg wrote: "error" would be just a single function that is available in the context parameter. There would be many others. From the DIP: "error" should also be able to specify where the error occurred in the input, so that it is displayed at the call site.
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 16:28:17 UTC, Timon Gehr wrote: I don't agree. (The argument against implicit mixin features was a complete lack of hygiene, not the mere possibility of accessing the caller scope.) How would you expect it to be hygienic in any way if it has implicit outer scope access? Reasoning is pretty much the same as with mixins.
Re: DIP 50 - AST macros
On 2013-11-11 17:28, Timon Gehr wrote: I don't agree. (The argument against implicit mixin features was a complete lack of hygiene, not the mere possibility of accessing the caller scope.) The macros system need to provide both hygienic and non-hygienic macros. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 2013-11-11 17:23, Timon Gehr wrote: "error" should also be able to specify where the error occurred in the input, so that it is displayed at the call site. That's a good point. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 13:26:37 UTC, Dicebot wrote: On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 I don't have time to investigate this in details but I'd like to mention that this is one of few possible new features where ROI is really worth it. It removes lot of burden from core language and allows to replace some very arcane template hacks with cleaner code. I'd say it is a bit too early to go for it (really, we need at least to figure out how to make releases without issues first :)) but it is a good future goal even in context of general feature freezing. Can you stop reading my mind, please ?
Re: DIP 50 - AST macros
On 11/10/2013 01:20 PM, Jacob Carlborg wrote: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 For macros that generate macros, I think you need a way to escape the splicing and maybe define how splicing works in an inner quasi quote. I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]> Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with the inner quasi quote, you'd have to do something like <[ <[ <[ \\\$exp ]> ]> ]> maybe a way to associate a splice with a quasi quote? ]> ]> a's exp is A b's exp is B c's exp is C then splice the quasi quote N times: 1: ast ]> 2: ast 3: ast A + B + C just dinking around here
Re: DIP 50 - AST macros
On 2013-11-11 20:23, Ellery Newcomer wrote: For macros that generate macros, I think you need a way to escape the splicing and maybe define how splicing works in an inner quasi quote. I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]> Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with the inner quasi quote, you'd have to do something like <[ <[ <[ \\\$exp ]> ]> ]> maybe a way to associate a splice with a quasi quote? ]> ]> a's exp is A b's exp is B c's exp is C then splice the quasi quote N times: 1: ast ]> 2: ast 3: ast A + B + C just dinking around here This would require some thought. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 19:23:21 UTC, Ellery Newcomer wrote: On 11/10/2013 01:20 PM, Jacob Carlborg wrote: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 For macros that generate macros, I think you need a way to escape the splicing and maybe define how splicing works in an inner quasi quote. I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]> Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with the inner quasi quote, you'd have to do something like <[ <[ <[ \\\$exp ]> ]> ]> maybe a way to associate a splice with a quasi quote? ]> ]> a's exp is A b's exp is B c's exp is C then splice the quasi quote N times: 1: ast ]> 2: ast 3: ast A + B + C just dinking around here $ refers to the enclosing scope. so $$foo should refers to $foo in the enclosing scope. No need for special rule or label.
Re: DIP 50 - AST macros
On 11/11/2013 05:47 PM, Dicebot wrote: On Monday, 11 November 2013 at 16:28:17 UTC, Timon Gehr wrote: I don't agree. (The argument against implicit mixin features was a complete lack of hygiene, not the mere possibility of accessing the caller scope.) How would you expect it to be hygienic in any way if it has implicit outer scope access? Reasoning is pretty much the same as with mixins. With mixins _every_ identifier is by _default_ inserted/looked up in the so-called 'outer scope', whereas declarations and normal symbol lookups within macros are hygienic by default. Macros are as much about reflection as about code generation. The trade-offs are simly not the same. Implicit outer scope access does not imply it behaves in a counter-intuitive manner. (It could just reflect some debug information after all, or perform some additional checks on the context for more safety, etc.) I will be really unhappy with any proposal that does not allow to interchange function calls with macro calls transparently, even though such a scheme allows more abusive code to be written as well. If in some code base macros should be clearly marked, a naming convention (eg. prefix macro names with 'macro') can be used. (Enforcing such a convention is trivial and amounts to eg. a few lines of additional code in a modified D front end.)
Re: DIP 50 - AST macros
On 11/11/2013 10:23 PM, Timon Gehr wrote: simly simply
Re: DIP 50 - AST macros
On 11/11/2013 12:06 PM, deadalnix wrote: On Monday, 11 November 2013 at 19:23:21 UTC, Ellery Newcomer wrote: On 11/10/2013 01:20 PM, Jacob Carlborg wrote: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 For macros that generate macros, I think you need a way to escape the splicing and maybe define how splicing works in an inner quasi quote. I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]> Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with the inner quasi quote, you'd have to do something like <[ <[ <[ \\\$exp ]> ]> ]> maybe a way to associate a splice with a quasi quote? ]> ]> a's exp is A b's exp is B c's exp is C then splice the quasi quote N times: 1: ast ]> 2: ast 3: ast A + B + C just dinking around here $ refers to the enclosing scope. so $$foo should refers to $foo in the enclosing scope. No need for special rule or label. so if I have to splice my ast N times, then I have to generate M $'s depending on when I want it to expand?
Re: DIP 50 - AST macros
On Monday, 11 November 2013 at 16:22:40 UTC, Timon Gehr wrote: On 11/11/2013 01:36 PM, Rikki Cattermole wrote: Yes outside of macros would be useful. For example code like this would become redundant: pragma(msg, "Support for x is not implemented on platform y"); static assert(0); Becoming: pragma(error, "Support for x is not implemented on platform y"); static assert(0, "Support for x is not implemented on platform y"); Hmm wasn't aware of that syntax. Needs an example I think on docs. I will still work on pull request as the point of an assert and pragma message is different, atleast I believe.
Re: DIP 50 - AST macros
On 11/10/13 11:46 PM, Jacob Carlborg wrote: On 2013-11-11 02:46, bearophile wrote: It's also useful to take a look at what F# is doing: http://tomasp.net/blog/2013/computation-zoo-padl/index.html I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#. There is significant regret about the way macros are defined in Scala. Probably not an example to follow. Andrei
Re: DIP 50 - AST macros
On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote: I've been thinking quite long of how AST macros could look like in D. I've been posting my vision of AST macros here in the newsgroup a couple of times already. I've now been asked to create a DIP out of it, so here it is: http://wiki.dlang.org/DIP50 Just a quick comment related to the 2007 proposal: http://youtu.be/FRfTk44nuWE?t=1h5m38s Page 45: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf Based on my read though, yours is more powerful. The 2007 proposal is basically the "Quasi-Quoting" portions of your proposal (which is such a small part of your proposal).
Re: DIP 50 - AST macros
On Tuesday, 12 November 2013 at 02:37:03 UTC, Andrei Alexandrescu wrote: On 11/10/13 11:46 PM, Jacob Carlborg wrote: On 2013-11-11 02:46, bearophile wrote: It's also useful to take a look at what F# is doing: http://tomasp.net/blog/2013/computation-zoo-padl/index.html I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#. There is significant regret about the way macros are defined in Scala. Probably not an example to follow. Do you have some article/documentation on that ? BTW, scala implemented macros as a plugin called macro paradise, which allowed people to play with it without crippling the core language. We should probably introduce new features that way.
Re: DIP 50 - AST macros
On Tuesday, 12 November 2013 at 02:37:03 UTC, Andrei Alexandrescu wrote: On 11/10/13 11:46 PM, Jacob Carlborg wrote: On 2013-11-11 02:46, bearophile wrote: It's also useful to take a look at what F# is doing: http://tomasp.net/blog/2013/computation-zoo-padl/index.html I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#. There is significant regret about the way macros are defined in Scala. Probably not an example to follow. Andrei Knowing what went wrong is important so as not to repeat past mistakes, is there more info somewhere to read on what went wrong? --rt
Re: DIP 50 - AST macros
On 11/11/13 9:16 PM, Rob T wrote: On Tuesday, 12 November 2013 at 02:37:03 UTC, Andrei Alexandrescu wrote: On 11/10/13 11:46 PM, Jacob Carlborg wrote: On 2013-11-11 02:46, bearophile wrote: It's also useful to take a look at what F# is doing: http://tomasp.net/blog/2013/computation-zoo-padl/index.html I'll do that. I've been looking at several languages, mainly Scala. But I have not looked at F#. There is significant regret about the way macros are defined in Scala. Probably not an example to follow. Andrei Knowing what went wrong is important so as not to repeat past mistakes, is there more info somewhere to read on what went wrong? Not yet. Martin mentioned to me in 2012 he doesn't think they took the right turns with defining macros. I don't know whether more information is available at the moment. Andrei
Re: DIP 50 - AST macros
On Tuesday, 12 November 2013 at 05:59:46 UTC, Andrei Alexandrescu wrote: Not yet. Martin mentioned to me in 2012 he doesn't think they took the right turns with defining macros. I don't know whether more information is available at the moment. Andrei Loosely related, but worth watching : http://www.infoq.com/presentations/data-types-issues
Re: DIP 50 - AST macros
On 2013-11-12 03:37, Andrei Alexandrescu wrote: There is significant regret about the way macros are defined in Scala. Probably not an example to follow. Unfortunately it's not easy to do anything about it if you cannot give us any more details to why. So far I haven't found anything. -- /Jacob Carlborg
Re: DIP 50 - AST macros
On 11/10/2013 1:20 PM, Jacob Carlborg wrote:
I've been thinking quite long of how AST macros could look like in D. I've been
posting my vision of AST macros here in the newsgroup a couple of times already.
I've now been asked to create a DIP out of it, so here it is:
http://wiki.dlang.org/DIP50
I confess I have some serious reservations about AST macros in general:
1. I've seen very heavy use of such macros in macro assemblers. What happens is
people use it to invent their own (very baroque) language on top of the existing
assembler one. Anyone trying to read the code has to learn this new unique
language, and given the limitations of the macro capability, it often comes with
unfixable bizarre corner cases and limitations.
This got so bad that in some cases I know of, the poor sap who is given the job
of making improvements resorted to running the assembler to generate an object
file, then disassembling the object file back into source code! Something went
very wrong for this to be necessary. I know in my own assembler work I have
abandoned all use of macros.
(Note that macros in typical assemblers are far more powerful than C's macro
language, which stands out for being pathetically underpowered.)
2. Macros are something that should be used very sparingly. However, this is not
what happens. I used to be perplexed that even top shelf C/C++ programmers tend
to write what I not so humbly consider to be pretty abusive use of macros. Now
it just saddens me.
3. Lisp is a language that encourages users to write macros to pretty much
invent a custom language for the task at hand. Heavy use of such makes the code
unrecognizable as being Lisp code. I believe a language ought to have some
"anchors" that the person reading the code can reliably recognize. (GO has taken
this idea pretty far.)
4. AST macros, pretty much by definition, manipulate the AST. However, if you
examine the semantic routines in DMD, a *lot* more is required to do more
significant things. The symbol table must be consulted, etc. I just don't see
how one could hope to implement something like function overloading with AST
macros. Or template type deduction. Or closure semantics, attribute inference,
etc. Sure, some forms of foreach can be rewritten into for statements by
manipulating the AST, but not the more interesting forms, such as the ones that
deal with opApply().
There are some statements in the DIP about the Context parameter and it'll
provide semantic information, but I don't see how this can work without making
it an ever-expanding collection of arbitrary methods.
5. We've said "no" in the past to things like user-defined tokens, which are
closely related.
6. Note that D has a limited form of AST macros with mixin templates.
To sum up, any AST macro proposal has some tough barriers to get over. Within
its necessarily severe semantic limitations, it has to deliver a pretty
compelling set of solutions, while avoiding the morass of incomprehensibility
that far too many macro systems become in practice.
For example, the Linq example in the DIP is not compelling, as aesthetically
nicer code can be written using D's ranges and algorithms:
auto data = arr.filter!(x => x > 5).array;
I see no compelling advantage in trying to make D code look like C#; to be blunt
it's like the old:
#define BEGIN {
#define END }
macros used in old C code to make it look like Pascal.
Re: DIP 50 - AST macros
I forgot to mention that "expression templates" can be used in D in an equivalent manner that they are used in C++, and can cover much of the functionality used in AST macros. Eric Anderton who wrote a regex expression template module a few years back: http://www.dsource.org/projects/ddl/browser/trunk/meta/regex.d
Re: DIP 50 - AST macros
On 11/12/2013 1:55 AM, Walter Bright wrote: I forgot to mention that "expression templates" can be used in D in an equivalent manner that they are used in C++, and can cover much of the functionality used in AST macros. Eric Anderton who wrote a regex expression template module a few years back: http://www.dsource.org/projects/ddl/browser/trunk/meta/regex.d Here's a project to add Linq support to C++: http://www.codeproject.com/Articles/17844/CLinq-LINQ-support-for-the-C-CLI-language There's also: http://pfultz2.github.io/Linq/ https://github.com/hjiang/linqxx/wiki
Re: DIP 50 - AST macros
On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright wrote: I forgot to mention that "expression templates" can be used in D in an equivalent manner that they are used in C++, They are crippled compared to C++, because you have no control over the return type of opCmp and opEquals, which means that you can't have expression templates involving comparisons. That's extremely limiting. My feeling with AST macros is: (1) AST macros are basically syntax sugar for string mixins. The one and only thing that string mixins have in their favour, is that they have almost no syntax. But that is actually a *huge* plus. Because it's syntax sugar, an AST macro syntax would be most convincing if you took some existing string mixins from various real projects, and show how beautiful they would become with macros. (2) The big functionality we don't have, is comprehensive reflection. I'd like to see more thought in that area. It seems challenging to define reflection in a way that doesn't expose compiler internals. The question I find particularly interesting is, "if we had macros, would reflection look different?"
Re: DIP 50 - AST macros
On Tuesday, 12 November 2013 at 09:20:07 UTC, Walter Bright wrote:
5. We've said "no" in the past to things like user-defined
tokens, which are closely related.
Regarding this point, I get that arbitrary expansion is out, but
would you be amenable to adding a small, _strictly-defined_ set
of operators that are compile-time errors unless overloaded?
#define BEGIN {
#define END }
macros used in old C code to make it look like Pascal.
A plague o' their houses!
-Wyatt
Re: DIP 50 - AST macros
On 2013-11-12 10:20, Walter Bright wrote:
For example, the Linq example in the DIP is not compelling, as
aesthetically nicer code can be written using D's ranges and algorithms:
auto data = arr.filter!(x => x > 5).array;
I was actually not the one that added the Linq examples. I like the
above better. I've used a similar example as well:
auto person = Person.where(e => e.name == "John");
Translates to:
select * from person where name = 'John'
But I don't see how that can currently be done in D. Operator
overloading in D isn't flexible enough. For example, when implementing
opEquals, you don't know if it's "==" or "!=" that's being called. Same
with the comparison operators.
I see no compelling advantage in trying to make D code look like C#; to
be blunt it's like the old:
#define BEGIN {
#define END }
macros used in old C code to make it look like Pascal.
I agree with this.
--
/Jacob Carlborg
