On 7/31/2011 9:56 PM, Jonathan M Davis wrote:
If we do a hand-written lexer of D for Phobos, it really should be a fairly
direct port of the dmd front-end. It should be _somewhat_ D-ified as
appropriate, (and the API should definitely be properly range-based and all
that), but the implementation needs to be fairly close to dmd itself so that
it's easy for someone to port changes and fixes back and forth between the two.
Otherwise, they're going to get out of sync fairly easily. If we're not going
to do a direct port, then we might as well just do the template-based lexer
generator that Andrei and others would really like to see (which we should
still do, but I think that the hand-written lexer is nowhere near as valuable
if it's not a direct port of dmd's lexer).

Yeah, I get the point, but I feel that I could port most trivial changes from DMD's lexer to lexer.d. The layout of the code is different, of course, but I'm borrowing most of the lexing logic from DMD. Plus with unittests, I think we could ensure things are lexing the same.

Also, I see _zero_ value in making it support D1. If it's for D2's standard
library, then what's the point of it lexing D1? That just complicates the
lexer for what is essentially a legacy product. And given that the differences
between D1 and D2 in dmd's lexer are covered with #ifdefs, it would be rather
complicated to try and do a direct port which covers both D1 and D2. It would
probably be easier if the two were completely separate.

Well, it doesn't appear that there are that many lexical differences between D1 and D2. A few operators, a few keywords, D2 supports a few different string constants, etc...

I wanted this to be usable by IDEs. Do we not want to support D2 for development? I could certainly make this D2 only.

As for what I've done so far, I'd have to go look. I haven't touched it in a
couple of months, I expect. There has been a lot of other stuff that I've
needed to do, and Andrei was trying to discourage such an implementation the
last time that I brought it up. So, I haven't exactly been in a rush to get it
done. I'd like to do it, but I've been rather busy.

Well, if Andrei wants to flex the power of D2's templates / etc... with a parser generator then maybe we should go down that route. Andrei, what do you think? Would the lexer/parser be generated at compile time or a a regular tool that would generate the appropriate D files?

So, if you really want to work on a potential D lexer for Phobos, that's fine,
but I really think that it needs to be a rather direct port, and that doesn't
sound like what you've been doing.

- Jonathan M Davis

If we decide to scrap my idea and go a more generic route, I'd probably start working on a standard database interface. I'd like to see D become a little more web framework friendly, and a nice generic database interface is definitely a start.

So, Andrei, what do you think about a lexer / parser in Phobos? Generic or a straight port?

Reply via email to