I now have a 'standalone' version of my DFA generator which can
be used to build lexers and run them at compile time.
This won't work with the current setup, because lexing is
all done first, then parsing. Well actually, on an include
file by include file basis recursively (note: include file
not #include file).
The lexed token stream is also passed through a number of filters
which are designed to read and write lists.
That non-trivial issue aside, there's a question what an extensible
lexer puts in user specified tokens. Something like:
Token of srcref * lexeme * constructor
makes sense: srcref is a fixed data type refering to the original
source location (file, line, column), lexeme is the actual string
lexed, and constructor is the name of the token as a string.
The problem is that with this system as written all GLR parser
productions would be much the same: Token Token Token ...
Even though we can check the constructor in the user action,
and 'Giveup' the parse if the wrong one is matched, this only
happens after the parse, in the user action.
Another solution is a fixed set of tokens like
Token1 of ..
Token2 of ..
Token3 of ...
and then chose one of them -- that reduces the spawned GLR parsers
by say 200 times, at the cost of having a finite set of tokens.
The restriction can probably be removed by hashing an infinite
integer onto a finite token and then ALSO do a check on the
user constructor. This is roughly what keyword lookup does now:
an identifier is lexed, then matched against a list of keywords,
and if found, a fixed token is returned in replacement.
--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Felix-language mailing list
Felix-language@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/felix-language
