On 9/11/2013 12:55 PM, H. S. Teoh wrote:
3. I assumed TokenType is a type. But it's not, it's an enum. Even
the document says it's a 'type', but it's not a type.
I don't understand this complaint. I thought it's pretty clear that it's
referring to what type of token it is (or would you prefer TokenKind?).
A type is a well-understood concept in D. This is using 'type' to mean something
completely different. This is very jarring. Your idea of using 'kind' is vastly
superior.
5. The LexerConfig initialization should be a constructor rather than
a sequence of assignments. LexerConfig documentation is awfully thin.
For example, 'tokenStyle' is explained as being 'Token style',
whatever that is.
I'm on the fence about this one. Setting up the configuration before
starting the lexing process makes sense to me.
Yes, using a constructor :-)
6. No clue how lookahead works with this. Parsing D requires arbitrary
lookahead.
What has this got to do with lexing?
The lexer needs to support backing up and going forward again. That's how the
dmd lexer works.
Also, it's unclear what types of input is supported -- the code example
only uses string input, but what about file input? Does it support
byLine? Or does it need me to slurp the entire file contents into an
in-memory buffer first?
The point of having the input be an InputRange is you DO NOT CARE if the input
is coming from a string, file, etc.
