Projects such as Pegged and our CTFE regex engine often serve as
poster-children of what is possible in D and many agree they are
among the more important projects.
I was thinking, after std.lexer is accepted, we have a stable
interface, but no matter how great the code is and even if it
beats the already superlative DMD lexer, it will _NOT_ be fast
during CTFE.
It is also often stressed in this very forum that it's paramount
for a lexer to be beyond fast.
For the above reasons, I propose that the compiler would offer an
interface to retrieve an already lexed buffer, similar in concept
to the existing token string q{...}
"Token strings open with the characters q{ and close with the
token }. In between must be valid D tokens."
By definition they contain only valid tokens... the compiler
would only have to create a range compatible with std.lexer...
