At 07:34 18/02/2009, Andreas Meyer wrote: >Let's say I have 600 keywords that I want to recognize as >something like an identifier, but in other places, I really >want different tokens. > >Currently, I have 600 rules that do some table lookup, so >that they consume an identifier and return one of the 600 >possible artificial keyword tokens. But still, this makes >the stream of tokens look like an endless stream of >identifiers to the parser, and I have a vague feeling >that this might be not the best possible solution.
You're right, that's going to make the parser slower. That's one of the reasons why I prefer the other approach to solve the "keywords as identifiers" problem -- recognise the keywords as individual tokens in the lexer, and then have a ridiculously long "identifier" rule in the parser that accepts the catch-all ID token in addition to all the tokens for the individual keywords. Since that's an integer match against a tokenset, it ought to be significantly faster than repeated string comparisons. (Having said that, I haven't actually *measured* this.) (It is slightly more work to add a new keyword, though, since you have to put it in two places.) List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en -~----------~----~----~----~------~----~------~--~---
