Walter Bright wrote:
1. It should accept as input an input range of UTF8. I feel it is a mistake to templatize it for UTF16 and UTF32. Anyone desiring to feed it UTF16 should use an 'adapter' range to convert the input to UTF8. (This is what component programming is all about.)
Why it is a mistake? I think Lexer should parse any UTF range and return compatible token's strings. That is it should provide strings for UTF8 input, wstrings for UTF16 input and so on.
