On Mon, Aug 6, 2012 at 8:03 PM, deadalnix <deadal...@gmail.com> wrote:
> The most complex thing that is needed is the policy to allocate identifiers > in tokens. It can be made by passing a function that have a string as > parameter and a string as return value. The default one would be an identity > function. I think one should pass it an empty symbol table and the lexer should fill it and associate each identifier with a unique ID, ID which would appear in the Identifier token. > The second parameter is a bool to tokenize comments or not. Is that enough ? > > The onError look like a typical use case for conditions as explained in the > huge thread on Exception. Yes, well we don't have a condition system. And using exceptions during lexing would most probably kill its efficiency. Errors in lexing are not uncommon. The usual D idiom of having an enum StopOnError { no, yes } should be enough.