On 7/31/2011 9:56 PM, Jonathan M Davis wrote:
If we do a hand-written lexer of D for Phobos, it really should be a fairly direct port of the dmd front-end. It should be _somewhat_ D-ified as appropriate, (and the API should definitely be properly range-based and all that), but the implementation needs to be fairly close to dmd itself so that it's easy for someone to port changes and fixes back and forth between the two. Otherwise, they're going to get out of sync fairly easily. If we're not going to do a direct port, then we might as well just do the template-based lexer generator that Andrei and others would really like to see (which we should still do, but I think that the hand-written lexer is nowhere near as valuable if it's not a direct port of dmd's lexer).
Yeah, I get the point, but I feel that I could port most trivial changes from DMD's lexer to lexer.d. The layout of the code is different, of course, but I'm borrowing most of the lexing logic from DMD. Plus with unittests, I think we could ensure things are lexing the same.
Also, I see _zero_ value in making it support D1. If it's for D2's standard library, then what's the point of it lexing D1? That just complicates the lexer for what is essentially a legacy product. And given that the differences between D1 and D2 in dmd's lexer are covered with #ifdefs, it would be rather complicated to try and do a direct port which covers both D1 and D2. It would probably be easier if the two were completely separate.
Well, it doesn't appear that there are that many lexical differences between D1 and D2. A few operators, a few keywords, D2 supports a few different string constants, etc...
I wanted this to be usable by IDEs. Do we not want to support D2 for development? I could certainly make this D2 only.
As for what I've done so far, I'd have to go look. I haven't touched it in a couple of months, I expect. There has been a lot of other stuff that I've needed to do, and Andrei was trying to discourage such an implementation the last time that I brought it up. So, I haven't exactly been in a rush to get it done. I'd like to do it, but I've been rather busy.
Well, if Andrei wants to flex the power of D2's templates / etc... with a parser generator then maybe we should go down that route. Andrei, what do you think? Would the lexer/parser be generated at compile time or a a regular tool that would generate the appropriate D files?
So, if you really want to work on a potential D lexer for Phobos, that's fine, but I really think that it needs to be a rather direct port, and that doesn't sound like what you've been doing. - Jonathan M Davis
If we decide to scrap my idea and go a more generic route, I'd probably start working on a standard database interface. I'd like to see D become a little more web framework friendly, and a nice generic database interface is definitely a start.
So, Andrei, what do you think about a lexer / parser in Phobos? Generic or a straight port?
