On Sunday 31 July 2011 22:28:51 Johann MacDonagh wrote: > On 7/31/2011 9:56 PM, Jonathan M Davis wrote: > > If we do a hand-written lexer of D for Phobos, it really should be a > > fairly direct port of the dmd front-end. It should be _somewhat_ > > D-ified as appropriate, (and the API should definitely be properly > > range-based and all that), but the implementation needs to be fairly > > close to dmd itself so that it's easy for someone to port changes and > > fixes back and forth between the two. Otherwise, they're going to get > > out of sync fairly easily. If we're not going to do a direct port, then > > we might as well just do the template-based lexer generator that Andrei > > and others would really like to see (which we should still do, but I > > think that the hand-written lexer is nowhere near as valuable if it's > > not a direct port of dmd's lexer). > > Yeah, I get the point, but I feel that I could port most trivial changes > from DMD's lexer to lexer.d. The layout of the code is different, of > course, but I'm borrowing most of the lexing logic from DMD. Plus with > unittests, I think we could ensure things are lexing the same. > > > Also, I see _zero_ value in making it support D1. If it's for D2's > > standard library, then what's the point of it lexing D1? That just > > complicates the lexer for what is essentially a legacy product. And > > given that the differences between D1 and D2 in dmd's lexer are covered > > with #ifdefs, it would be rather complicated to try and do a direct > > port which covers both D1 and D2. It would probably be easier if the > > two were completely separate. > > Well, it doesn't appear that there are that many lexical differences > between D1 and D2. A few operators, a few keywords, D2 supports a few > different string constants, etc... > > I wanted this to be usable by IDEs. Do we not want to support D2 for > development? I could certainly make this D2 only. > > > As for what I've done so far, I'd have to go look. I haven't touched it > > in a couple of months, I expect. There has been a lot of other stuff > > that I've needed to do, and Andrei was trying to discourage such an > > implementation the last time that I brought it up. So, I haven't > > exactly been in a rush to get it done. I'd like to do it, but I've been > > rather busy. > > Well, if Andrei wants to flex the power of D2's templates / etc... with > a parser generator then maybe we should go down that route. Andrei, what > do you think? Would the lexer/parser be generated at compile time or a a > regular tool that would generate the appropriate D files? > > > So, if you really want to work on a potential D lexer for Phobos, that's > > fine, but I really think that it needs to be a rather direct port, and > > that doesn't sound like what you've been doing. > > > > - Jonathan M Davis > > If we decide to scrap my idea and go a more generic route, I'd probably > start working on a standard database interface. I'd like to see D become > a little more web framework friendly, and a nice generic database > interface is definitely a start. > > So, Andrei, what do you think about a lexer / parser in Phobos? Generic > or a straight port?
I can tell you right now that he wants a generic, template-based lexer / parser generator rather than a hand-written solution. He doesn't like the idea of the hand-written solution. And I definitely think that we should have a template-based lexer / parser generator. The question is whether it's also worth having a hand-written port of the dmd front-end's lexer and parser to Phobos so that we can have on official lexer and parser in Phobos which parse D the same way that the compiler does. - Jonathan M Davis
