Re: std.d.lexer: pre-voting review / discussion

[Timon Gehr]

On 09/12/2013 06:40 PM, H. S. Teoh wrote:

I vote for Tok!"..." to denote token types.

Question: what's the implementation of Tok?  Does it fit into an enum?
What's the underlying representation? I imagine some kind of canonical
mapping into an integral type would be desired, to maximize runtime
performance.


This is just a quick hack. One would maybe want to avoid the unwarranted 
copies and linear searches at compile time:

import std.algorithm;

private enum symbols = ["i",".","..",",","0",/+...+/];

struct TokenType{
    private uint _tag; // or smaller type
    private this(int tag){_tag=tag;}
    string toString(){ // (optional)
        static array(R)(R s){ string[] r; foreach(x;s) r~=x; return r; }
        static immutable strs=array(symbols.map!((a)=>`Tok!"`~a~`"`));
        return strs[_tag];
    }
}

template Tok(string name)if(symbols.countUntil(name)!=-1){
    enum Tok=TokenType(cast(uint)symbols.countUntil(name));
}

import std.stdio;
void main(){
    enum x = Tok!"i";
    writeln(x);
    writeln(Tok!"0");
}