On 14.05.2012 21:16, Tobias Pankrath wrote:
struct Token{
uint col, line;
uint flags;//indicated info about token, serves as both type tag and
flag set;
//indicates proper type once token was cooked (like "31.415926" ->
3.145926e1) i.e. values are calculated
union {
string chars;
float f_val;
double d_val;
uint uint_val;
long long_val;
ulnog ulong_val;
//... anything else you may need (8 bytes are plenty)
}//even then you may use up to 12bytes
//total size == 24 or 20
};
I wouldn't convert the string representation to a value unless needed.
Even the parser does not need to know.
Thx for snipping out the gist of it :)
See last sentence, flexibility is the king:
<<<
Where:
Each raw token at start has chars == slice of characters in text
(or if not UTF-8 source = copy of source). Except for keywords and
operators.
Cooking is a process of calculating constant values and such (say
populating symbols table will putting symbol id into token instead of
leaving string slice). Do it on the fly or after the whole source - let
the user choose.
>>>
--
Dmitry Olshansky
