On Wed, Jun 17, 2009 at 21:46:46 +0200, [email protected] wrote: > > The easiest way to implement it in the token reader would probably > be to replace the "reader->state = X" statements with calls to a > private function (e.g. enter_state) that would also store the > token's starting position. Then in flush_token, store this value > when emitting a token. > > Many thanks for the hints :) > > I just merged the type4+tokeniser patch in the trunk. All the unit > tests for the type 4 functions are passing.
The code you committed changes beg_pos in flush_token even if no token is emitted, which could leave it pointing at an unexpected position after an error like EAGAIN. I think that line of code belongs between "*token = new_tok;" and "finish:". -- Michael
signature.asc
Description: Digital signature
