Bob Frankel wrote: > following idioms i've seen posted elsewhere, i've created a MyToken > class that inherits from CommonToken; my Lexer then overrides Token > Lexer.emit(), at which time i create instances of MyToken.... > > things work fine, until the parser encounters a syntactic error.... > from what i can tell, the parser inserts an "error" token of type > CommonToken; this then leads to a class cast exception in the > surrounding parser rule when attempting to assign a CommonToken value > through a generated (MyToken) cast....
There are two possible causes of this: a) There was a bug in ANTLR up to version 3.1.3 where the generated code would sometimes create tokens using 'new CommonToken', even when the TokenLabelType option is set to MyToken. See the thread at <http://www.antlr.org/pipermail/antlr-interest/2009-July/thread.html#35129>. I think this has been fixed in ANTLR 3.2 (although not in the way I suggested that would also have fixed point b) below). Note your token class will need to have the same constructors as CommonToken in order for the fix to work. b) When the getMissingSymbol method of a parser inserts a token in order to recover from an error, the inserted token is of type CommonToken. Override it as follows (this implementation also fixes a different bug that can cause a NullPointerException): @parser::members { /** * Work around an ANTLR bug that causes a NullPointerException when trying * to recover at the end of the input stream. Also ensure that inserted * tokens are of type MyToken. */ @Override protected Object getMissingSymbol(IntStream input, RecognitionException re, int expectedTokenType, BitSet follow) { String tokenText = null; if (expectedTokenType == Token.EOF) { tokenText = "<missing EOF>"; } else if (expectedTokenType >= 0 && expectedTokenType < getTokenNames().length) { tokenText = "<missing " + getTokenNames()[expectedTokenType] + ">"; } else { throw new Error("invalid expectedTokenType " + expectedTokenType); } MyToken t = new MyToken(expectedTokenType, tokenText); Token current = ((TokenStream) input).LT(1); if (current == null || current.getType() == Token.EOF) { current = ((TokenStream) input).LT(-1); } if (current != null) { // If there are any other position-related fields in your MyToken // class, set them here. t.setLine(current.getLine()); t.setCharPositionInLine(current.getCharPositionInLine()); } t.setChannel(DEFAULT_TOKEN_CHANNEL); return t; } } -- David-Sarah Hopwood ⚥ http://davidsarah.livejournal.com
signature.asc
Description: OpenPGP digital signature
List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
