On Sun, Jul 08, 2012 at 09:59:38AM +0200, Mehrdad wrote: [...] > Right, it's trivial to fix with an extra state variable like > 'consumedtrailingdot'.
This is eventually what I did in my own D lexer. Well, actually, I kinda blasted an ant with an M16... I had a queue of backlogged tokens which getNext will return if non-empty, and when I recognized things like 1..4, I would push 2 or 3 tokens onto the backlog queue, so no extra state is needed (although the backlog queue itself is really just another form of extra state). > The point was, it requires an extra lookahead character, which I was > trying to avoid (mainly for fun). > > In this case, it doesn't really make a difference in practice -- but > in general I don't like lookaheads, because depending on future data > makes it hard for e.g. the user to enter data via the console. In my case, the extra lookahead only happens when the lexer sees string prefixes like "4..", which doesn't usually happen at the end of a line. In all other cases, no lookahead is actually necessary, so except for the very rare case, entering data via console actually works just fine. T -- I am a consultant. My job is to make your job redundant. -- Mr Tom
