Re: Let's stop parser Hell

[Jacob Carlborg]

On 2012-08-01 00:38, Jonathan M Davis wrote:

I don't have the code with me at the moment, but I believe that the token type
looks something like
struct Token
{
  TokenType type;
  string str;
  LiteralValue value;
  SourcePos pos;
}

struct SourcePos
{
  size_t line;
  size_ col;
  size_t tabWidth = 8;
}


What about the end/length of a token? Token.str.length would give the 
number of bytes (code units?) instead of the number of characters (code 
points?). I'm not entirely sure what's needed when, for example, doing 
syntax highlighting. I assume you would know the length in characters of 
a given token internally inside the lexer?

--
/Jacob Carlborg