Indhu, Thanks for that. I'll keep my tokenStream.LT(0) cludge for now. ITokenStream doesn't have a getTokens() method, and changing my code to reference CommonTokenStream instead of ITokenStream breaks my unit tests that use mock objects.
FWIW, as a relative newcomer to ANTLR, it seems to me that either CommonTokenStream isn't correctly honouring the ITokenSteam interface, or else perhaps ITokenStream ought to be updated to formalise CommonTokenStream's behaviour (though I'm not familiar enough with the other ITokenStream sub-classes to be sure about this). Cheers, Chris 2009/4/16 Indhu Bharathi <[email protected]> > This is expected behavior only. It is designed this way so that user can > filter for a particular channel if needed. For your task, you can use > getTokens() method which will return a List of tokens that can be iterated. > > Cheers, Indhu > > Chris Lambrou wrote: > > Yesterday I was stung by some odd behaviour in CommonTokenStream, whereby I > was trying to iterate over the token stream looking for tokens of a specific > type. Since ITokenStream doesn't implement IEnumerable, it appears that the > way to do this is as follows: > > for (int i = 0; i < tokenStream.Size(); i++) > { > IToken token = tokenStream.Get(i); > > //... do stuff with token... > } > > However, I was finding that tokenStream.Size() returned 0, despite my token > stream being non-empty. It seems that the underlying stream is lazily > populated internally, and CommonTokenStream.Size() doesn't trigger a load. I > had to invoke tokenStream.LT(0) to trigger the lazy load prior to looping > through the tokens. Is this intended behaviour? Does is happen in all of the > different runtimes? If so, it's very counterintuitive. > > Incidentally, could ITokenStream be updated to implement > IEnumerable<IToken> please? Would others find this useful? I don't mind > doing the work, but to whom should I submit a patch? > > Chris > > ------------------------------ > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: > http://www.antlr.org/mailman/options/antlr-interest/your-email-address > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en -~----------~----~----~----~------~----~------~--~---
List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
