As we all know, tool support is important for D's success. Making tools easier
to build will help with that.
To that end, I think we need a lexer for the standard library - std.lang.d.lex.
It would be helpful in writing color syntax highlighting filters, pretty
printers, repl, doc generators, static analyzers, and even D compilers.
It should:
1. support a range interface for its input, and a range interface for its output
2. optionally not generate lexical errors, but just try to recover and continue
3. optionally return comments and ddoc comments as tokens
4. the tokens should be a value type, not a reference type
5. generally follow along with the C++ one so that they can be maintained in
tandem
It can also serve as the basis for creating a javascript implementation that can
be embedded into web pages for syntax highlighting, and eventually an
std.lang.d.parse.
Anyone want to own this?
