On 2012-08-01 00:38, Jonathan M Davis wrote:
I don't have the code with me at the moment, but I believe that the token type looks something likestruct Token { TokenType type; string str; LiteralValue value; SourcePos pos; } struct SourcePos { size_t line; size_ col; size_t tabWidth = 8; }
What about the end/length of a token? Token.str.length would give the number of bytes (code units?) instead of the number of characters (code points?). I'm not entirely sure what's needed when, for example, doing syntax highlighting. I assume you would know the length in characters of a given token internally inside the lexer?
-- /Jacob Carlborg
