On Sunday, 27 January 2013 at 09:51:10 UTC, Brian Schott wrote:
I'm writing a D lexer for possible inclusion in Phobos.
DDOC:
http://hackerpilot.github.com/experimental/std_lexer/phobos/lexer.html
Code:
https://github.com/Hackerpilot/Dscanner/blob/range-based-lexer/std/d/lexer.d
It's currently able to correctly syntax highlight all of
Phobos, but does a fairly bad job at rejecting or notifying
users/callers about invalid input.
I'd like to hear arguments on the various ways to handle errors
in the lexer. In a compiler it would be useful to throw an
exception on finding something like a string literal that
doesn't stop before EOF, but a text editor or IDE would
probably want to be a bit more lenient. Maybe having it
run-time (or compile-time configurable) like std.csv would be
the best option here.
I'm interested in ideas on the API design and other high-level
issues at the moment. I don't consider this ready for
inclusion. (The current module being reviewed for inclusion in
Phobos is the new std.uni.)
Having a "standard" lexer module, that many people work on, and
has up-to-date rules, is priceless! Thank you for doing this.
