On 8/3/2012 6:18 AM, deadalnix wrote:
lexer can have a parameter that tell if it should build a table of token or slice the input. The second is important, for instance for an IDE : lexing will occur often, and you prefer slicing here because you already have the source file in memory anyway.
A string may span multiple lines - IDEs do not store the text as one string.
If the lexer allocate chunks, it will reuse the same memory location for the same string. Considering the following mecanism to compare slice, this will require 2 comparaisons for identifier lexed with that method : if(a.length != b.length) return false; if(a.ptr == b.ptr) return true; // Regular char by char comparison. Is that a suitable option ?
You're talking about doing for strings what is done for identifiers - returning a unique handle for each. I don't think this works very well for string literals, as there seem to be few duplicates.
