On 09/12/2013 08:17 AM, Walter Bright wrote:
I don't believe that, because you can see about anything for tokens in lookahead and so have to duplicate nearly the whole lexer anyway for the 'fast path', but you're free to try it out and prove me wrong.
Me neither and if you have to duplicate the code for this it's a double loss. I think it would be more fruitful to batch lexing, i.e. fill a hole memory page (4Kb) with tokens the go back to parsing until you need more.
