Thank you! That's exactly what I needed to figure out how to do, and the extra code to override emit() and nextToken() were godsends.
On Mon, May 2, 2011 at 8:50 AM, Bart Kiers <[email protected]> wrote: > On Mon, May 2, 2011 at 1:19 AM, Todd O'Bryan <[email protected]> wrote: > >> ... >> >> >> Does this make any sense? Is there some way to deal with it? >> ... > > > You could let '/]]' be matched in the 'R_TAG' rule and emit another token as > per the instructions described here: > http://www.antlr.org/wiki/pages/viewpage.action?pageId=3604497 > > A demo: > > lexer grammar TLexer; > > @members { > > List<Token> tokens = new ArrayList<Token>(); > > private void emit(String text, int type) { > Token token = new CommonToken(type, text); > token.setType(type); > emit(token); > } > > @Override > public void emit(Token token) { > state.token = token; > tokens.add(token); > } > > @Override > public Token nextToken() { > super.nextToken(); > if(tokens.size() == 0) { > return Token.EOF_TOKEN; > } > return (Token)tokens.remove(0); > } > } > > L_TAG > : '[/' > ; > > R_TAG > : '/]]' {emit("/", ANY); emit("]]", R_BRACKET);} > | '/]' > ; > > L_BRACKET > : '[[' > ; > > R_BRACKET > : ']]' > ; > > SPACE > : (' ' | '\t' | '\r' | '\n') {skip();} > ; > > ANY > : . > ; > > which can be tested with the class: > > import org.antlr.runtime.*; > > public class Main { > public static void main(String[] args) throws Exception { > String source = "[/ foo /] [[/ bar /]]"; > ANTLRStringStream in = new ANTLRStringStream(source); > TLexer lexer = new TLexer(in); > CommonTokenStream tokens = new CommonTokenStream(lexer); > for(Object o : tokens.getTokens()) { > Token t = (Token)o; > System.out.println("text=" + t.getText() + ", type=" + t.getType()); > } > } > } > > > Regards, > > Bart. > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: > http://www.antlr.org/mailman/options/antlr-interest/your-email-address > List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
