Just thinking out loud here. Would it be possible to lex a file in parallel? Cutting it in half (or similar) and lex both pieces simultaneously in parallel.
I think a lexer should be IO-bound on todays machines, so parallelizing should not give much benefits.
Of course, you can write a microbenchmark, such that lexing is compute-bound and parallelizing gives a speedup. ;)
