On Monday, 14 May 2012 at 17:05:17 UTC, Dmitry Olshansky wrote:
On 14.05.2012 19:10, Roman D. Boiko wrote:
(Subj.) I'm in doubt which to choose for my case, but this is
a generic
question.
http://forum.dlang.org/post/[email protected]
Cross-posting here. I would appreciate any feedback. (Whether
to reply
in this or that thread is up to you.) Thanks
On 14.05.2012 19:10, Roman D. Boiko wrote:
Oops, sorry I meant to post to NG only :) Repost:
Clearly you are puting too much pressure on Token.
In my mind it should be real simple:
struct Token{
uint col, line;
uint flags;//indicated info about token, serves as both
type tag and flag set;
//indicates proper type once token was cooked (like "31.415926"
-> 3.145926e1) i.e. values are calculated
union {
string chars;
float f_val;
double d_val;
uint uint_val;
long long_val;
ulnog ulong_val;
//... anything else you may need (8 bytes are plenty)
}//even then you may use up to 12bytes
//total size == 24 or 20
};
Where:
Each raw token at start has chars == slice of characters in
text (or if not UTF-8 source = copy of source). Except for
keywords and operators.
Cooking is a process of calculating constant values and
such (say populating symbols table will putting symbol id into
token instead of leaving string slice). Do it on the fly or
after the whole source - let the user choose.
Value types have nice property of being real fast, I suggest
you to do at least some syntetic tests before going with
ref-based stuff. Pushing 4 word is cheap, indirection never is.
Classes also have hidden mutex _monitor_ field so using 'class'
can be best described as suicide.
Yet you may go with freelist of tokens (leaving them as
structs). It's an old and proven way.
About row/col - if this stuff is encoded into Finite Automation
(and you sure want to do something like DFA) it comes out
almost at no cost.
The only disadvantage is complicating DFA tables with some
irregularities of "true Unicode line ending sequences". It's
more of nuisance then real problem though.
P.S. if you real bend on performance, I suggest to run sparate
Aho-Corassic-style thing for keywords, it would greatly
simplify (=speed up) DFA structure if keywords are not
hardcoded into automation.
Thanks Dmitry, I think I'll need to contact you privately about
some details later.
