On 10/8/13 6:26 PM, Walter Bright wrote:
On 10/4/2013 5:24 PM, Andrei Alexandrescu wrote:
[...]
Some points:
1. This is a replacement for the switch statement starting at around
line 505 in advance()
https://github.com/Hackerpilot/phobos/blob/9bdb7f97bb8021f3b0d0291896b8fe21a6fead23/std/d/lexer.d
It is not a replacement for the rest of the lexer.
2. Instead of explicit token type enums, such as:
mod, /// %
it would just be referred to as:
tok!"%"
Andrei pointed out to me that he has fixed the latter so it resolves to
a small integer - meaning it works efficiently as cases in switch
statements. This removes my primary objection to it.
3. This level of abstraction combined with efficient generation cannot
be currently done in any other language. Hence, it makes for a sweet
showcase of what D can do.
Hence, I think we ought to adapt Brian's lexer by replacing the switch
with Andrei's trie searcher, and replacing the enum TokenType with the
tok!"string" syntax.
Thanks, that's exactly what I had in mind. Also the trie searcher should
be exposed by the library so people can implement other languages.
Let me make another, more strategic, point. Projects like Rust and Go
have dozens of people getting paid to work on them. In the time it takes
us to crank one conventional lexer/parser for a language, they can crank
five. The answer is we can't win with a conventional approach. We must
leverage D's strengths to amplify our speed of execution, and in this
context an integrated generic lexer generator is the ticket.
There is one thing I neglected to mention, and I apologize for that.
Coming with this all on the eve of voting must be quite demotivating for
Brian, who's been through all the arduous steps to get his work to
production quality. I hope the compensating factor is that the proposed
change is a net positive for the greater good.
Andrei
