Jim Idle wrote: > David-Sarah Hopwood wrote: >> Jim Idle wrote: >>> David-Sarah Hopwood wrote: >>>> Gavin Lambert wrote: >>>> >>>>> At 10:12 14/08/2009, [email protected] wrote: >>>>> >QuizLexer lexer = new QuizLexer(new ANTLRFileStream(input)); >>>>> >CommonTokenStream tokens = new CommonTokenStream(lexer); >>>>> >// prints 0 >>>>> >System.out.println(tokens.size()); >>>>> > >>>>> >So according to this there are no tokens, which would explain >>>>> >the blank debugger Input window. If the ANTLRWorks Interpreter runs >>>>> >the code correctly, why not the Debugger or unit tests? >>>>> >>>>> Actually that's a bit deceptive :) >>>>> >>>>> The stream starts out empty, and thus .size() returns 0. >>>>> >>>> You're right, and that seems like a bug. In CommonTokenStream: >>> >>> No - you need to calls something like LT(1) first - otherwise all methods >>> have >>> to check for init which is an overhead that isn't really needed. >> >> Except that all public methods except .size() and .consume() already do >> (directly or indirectly) perform this check. That is, almost all of the >> overhead is being paid already -- and is needed, since the generated >> parser code doesn't ensure that it calls LT, or any other single method, >> first. > > Parser calls token stream, it checks for empty stream, then fills buffer.
Yes, and that check is implemented by testing for p == -1 in all of the public methods of CommonTokenStream except .size() and .consume(). The omission of .size() and .consume() is a bug. Adding the check to .size() wouldn't have any significant performance impact, since it's not a frequently called method. Not adding it to .consume() on performance grounds would be premature optimization. (There are obvious optimizations to .consume() and .skipOffTokenChannels(int) that would more than offset this, *if* profiling showed that it were important to do so.) -- David-Sarah Hopwood ⚥ http://davidsarah.livejournal.com List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en -~----------~----~----~----~------~----~------~--~---
