Hi Michael. I've attached some files that partially implement a tokeniser for PDF files. If there's interest, I can clean it up for inclusion.
Very nice! Thanks. I will take a deeper look at it as soon as possible. In the meanwhile, did you try to make the tokeniser to use a reading stream? Would be a nice try out of the get_char and peek_char operations in the streams.
