On 09/11/2013 05:01 PM, Dicebot wrote:
std.d.lexer is standard module for lexing D code, written by Brian Schott
Nice!
Please take this part seriously: "If you identify problems along the
way, please note if they are minor, serious, or showstoppers."
(http://wiki.dlang.org/Review/Process). This information later
will be used to determine if library is ready for voting.
I only had a short look at the source code and was left with a good
impression, i.e. it looks like a lexer written in D.
I do like how you made the lexer configurable to serve the different
needs. Also it was very easy to setup and use.
The appended underscore for keywords is much nicer than messing with
uppercase/lowercase which was what I did.
One thing that bothered me was the usage of const(ubyte)[] as input.
If you operate on UTF-8 it should take const(char)[] you can cast it to
const(ubyte)[] internally.
Also a convenience function that reads a file and processes UTF BOM
marks would be nice (see toUtf8
https://github.com/dawgfoto/lexer/blob/master/dlexer/dlexer.d#L1429),
but that could as well fit somewhere else into phobos.
---- Information request from module author ----
Performance was a major discussed topic in previous thread. Could you
please provide benchmarking data for version currently ongoing the review?
Nice too, it works as fast as my D lexer
(https://github.com/dawgfoto/lexer) which is about as fast as DMD's
lexer (when compiled with LDC).
