On 10/6/13 2:10 AM, Jacob Carlborg wrote:
On 2013-10-05 02:24, Andrei Alexandrescu wrote:
Such a trie searcher is not intelligent, but is very composable and
extremely fast. It is just smart enough to do maximum munch (e.g.
interprets "==" and "foreach" as one token each, not two), but is not
smart enough to distinguish an identifier "whileTrue" from the keyword
"while" (it claims "while" was found and stops right at the beginning of
"True" in the stream). This is for generality so applications can define
how identifiers work (e.g. Lisp allows "-" in identifiers but D doesn't
etc). The trie finder doesn't do numbers or comments either. No regexen
of any kind.
Would it be able to lex Scala and Ruby? Method names in Scala can
contain many symbols that is not usually allowed in other languages. You
can have a method named "==". In Ruby method names are allowed to end
with "=", "?" or "!".
Yes, easily. Have the trie matcher stop upon whatever symbol it detects
and then handle the tail with Ruby-specific code.
Andrei
