I have an ANTLR application with which I process a large (100Mb) file by
means of a filter grammar (@options{filter=true;}), in search of a simple
pattern. No backtracking is necessary. Each time the pattern is found, a
call is made to a (C#) method that does some bookkeeping.
The file is scanned with the following loop:
while (lexer.NextToken () != Token.EOF_TOKEN) { }
When the pattern is encountered, the C# method is called. From 'the book' at
section 5.8 (filter option) I understood that 'the lexer yields an
incomplete stream of tokens'. This, however, does not prevent an out of
memory exception to occur after a while.
How can I prevent this from happening? No backtracking whatsoever needs to
occur, so I do not need a token history to be retained during the execution
of the above loop. I have tried inserting {Skip();} statements, but this
does not seem to help.
I noticed that the exception does not occur (and scanning the file goes
considerably faster) when in lexer.NextToken() I comment out
int m = input.Mark();
and
input.Rewind(m);
but I am not sure what undesired effect this may have.
Regards, John Pool
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
