At 06:02 11/09/2009, Sylvain, Gregory [USA] wrote: >I have a bunch of questions about how ANTLR (v3) is lexing it's >input stream. I am continually chasing bugs about how ANTLR >lexed some text as one token when I was expecting it to Lex it as >another token. > >I've checked out this list and elsewhere on the ANTLR.org site >for references to lexical analysis and how ANTLR is doing it, but >I have not found much. > >Specifically, I am trying to understand the order of evaluation >of the lexer rules and how ANTLR combines them in order complete >the analysis of it's given input.
The best thing to do is to write a bunch of unit tests for your lexer. Not only is this a good sanity check that you're getting the output you want, it's a safety net that will help ensure that adding or modifying rules don't break existing examples. The first thing to realise is that the lexer runs independently, without any context from the parser. So if you have two lexer rules that could potentially match the same input, then which ones you have used in the parser are not going to help disambiguate between them. The general rule of thumb that ANTLR follows when deciding between lexer tokens is that longest-match wins; failing that, whichever rule is listed first will win. Best practice is to ensure that your lexer rules are completely unambiguous; stick to generalities and don't try to assign semantic meaning until at least the parser stage. It's also a good idea to merge rules that have a common left prefix (and "left factor" them) to reduce the amount of lookahead required to disambiguate between them, and also to give you more control over the disambiguation if you need it. A classic example of this (on the wiki) is INT vs. FLOAT vs. RANGE. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en -~----------~----~----~----~------~----~------~--~---
