Like Aaron said, you'd just parse over a stream of tokens. Autumn supports this as follow: each parsing expression (henceforth: "parser") receives a "Parse" object when invoked. This Parse object contains either a string input or an object list input, along with the current position in whichever input is being used. The primitive parsers are specialized on an input type (matching text or tokens) but all the higher-level combinators (sequences, choices) don't care, as they don't lookup the input directly. Pretty easy, nothing to write home about.
Nicolas LAURENT On Sat, 11 May 2019 at 00:21, Matthew <matt...@scratchy.org.nz> wrote: > I can't point you to any reading but my PEG parser generator works (and > requires) a separate tokenizing stage (it just, personally, made more sense > to me). The tokenizer is built in, but it was designed to allow an external > tokenizer. (Generates only Java code though). > > Matthew > On 11/05/19 10:01 AM, Juancarlo Añez wrote: > > Hello, > > Please point me to the reading list for making PEG work when there's an > existing and required tokenizer? > > Thanks in advance, and regards, > > -- > Juancarlo *Añez* > > _______________________________________________ > PEG mailing > list...@lists.csail.mit.eduhttps://lists.csail.mit.edu/mailman/listinfo/peg > > _______________________________________________ > PEG mailing list > PEG@lists.csail.mit.edu > https://lists.csail.mit.edu/mailman/listinfo/peg >
_______________________________________________ PEG mailing list PEG@lists.csail.mit.edu https://lists.csail.mit.edu/mailman/listinfo/peg
