On 8/1/2011 5:50 PM, Jonathan M Davis wrote:
On 7/31/2011 11:32 PM, Walter Bright wrote:
On 7/31/2011 6:29 PM, Johann MacDonagh wrote:
Basically, you give it some string (string, wstring, or dstring), and
it gives
you a range of tokens back. The token has the type, a slice of the
input that
corresponds to the token, line / column, and a value (e.g. an integer
constant).
I suggest making the input a range.
So, the reason I didn't do this initially was to make slicing a lot
easier. It could easily be changed to any kind of input range of
characters. However, it looks like something that's not a direct port of
DMD isn't worthwhile.
Some adjustment can be made. It _does_ need to have a range-based API, and
some changes are inevitable in order to do that. But the changes to the basic
logic need to be minimal.
The thing is that it can be an input range _and_ take advantage of slicing
with the use of static ifs in the appropriate places. If hasSlicing!Range is
true, then you can have a static if branch for slicing it. If it's a narrow
string then you can use another branch. And if it's neither, then it won't get
the advantages of slicing. It complicates the code somewhat, but the proper
use of Phobos functions which deal with it for you should help.
In any case, the point is that we need to port dmd's lexer over to D, changing
it enough to give it a range-based interface (so it'll take a range of dchars
but return a range of tokens) but avoiding changing more than necessary so
that it's easy to port changes and fixes back and forth between it and dmd's
front end.
- Jonathan M Davis
Ok, I'll work on that this weekend. I'll do as close to a port as
possible while stripping out anything that doesn't make sense and
D-ifying the code a little bit. It will accept anything that
isInputRange!R && isSomeChar!(ElementEncodingType!R) and return an input
range of tokens. I've already looked through quite a bit of the lexer.c
source while writing my other lexer, and the port should be easy (famous
last words).
