On 1/16/2016 7:13 AM, H. S. Teoh via Digitalmars-d wrote:
I disagree. I think having the dmd itself (lexer, parser, etc.) as a
library (with the dmd executable merely being the default frontend) will
do D a lot of good.
For one thing, IDE's will no longer need to reinvent a D parser for the
purposes of syntax highlighting;
On the other hand, using lexer.d and parse.d as a guide to build your own is a
trivial undertaking. The Boost license is designed so this can be done without
worrying about making a derived work.
I looked into doing syntax highlighting for my editor, MicroEmacs. It turns out
it is not so easy to just use a compiler lexer/parser for it. For one thing, the
one used in the compiler is optimized for speed in a forward pass through the text.
But a syntax highlighter in a text editor is different. Suppose I change a
character in the middle of a line. All the highlighting from that point forward
may change. And to figure out what that change is, the parser/lexer has to start
over from the beginning of the file! (Think string literals, nested comments,
quoted string literals, etc.) This would make editing slow.
I surmised that a solution is to have each line in the editor be tagged with a
state to show the lexing state at the beginning of that line. Then, when a
character in a line changes, the lexer can be restarted from that state, and it
continues forward until the next line state matches the new computed state. This
would make it enormously faster.
But the compiler's lexer is not designed to be restartable in the middle.
Similarly, a source code formatter would be different in a different way. It
would need, for example, extra information about comments, token start/end
positions, etc. Such data collection would be irrelevant to a compiler, and
would slow it down and consume unneeded memory.
