Andrei Alexandrescu Wrote: > > I have in mind the entire implementation of a simple design, but never > had the time to execute on it. The tokenizer would work like this: > > alias Lexer!( > "+", "PLUS", > "-", "MINUS", > "+=", "PLUS_EQ", > ... > "if", "IF", > "else", "ELSE" > ... > ) DLexer; > > Such a declaration generates numeric values DLexer.PLUS etc. and > generates an efficient code that extracts a stream of tokens from a > stream of text. Each token in the token stream has the ID and the text.
What about, say, floating-point literals? It seems like the first element of a pair might have to be a regex pattern.
