"Walter Bright" <newshou...@digitalmars.com> wrote in message news:ia5j41$2bn...@digitalmars.com... > > To specifically answer your question, yes, in the lexers I make, you know > you're parsing a string, so you process it as you parse it. > >... > > I don't know. I'd have to study the issue for a while. I suggest taking a > look at dmd's lexer and compare. I'm not sure what Spirit's approach to > this is. >
I've taken a deeper look at Spirit's docs: In the older Spirit 1.x, the lexing is handled as part of the parsing. The structure of it definitely suggests it should be easy for it to do all token-conversion right as the string is being lexed, although I couldn't tell whether or not it actually did so (I'd have to look at the source). But, since Spirit 1.x doesn't handle lexing separately from parsing, I *think* backtracking (it *is* a backtracking parser) results in re-lexing, even for terminals that never get special processing, such as keywords (But I'm not completely certain because I don't have much experience with LL). In Spirit 2.x, standard usage involves having the lexing separate from parsing. I didn't see anything at all in the docs for Spirit 2.x that seemed to suggest even the possibility of it processing tokens as they're lexed. However, Spirit is designed with heavy policy-based customizability in mind, so such a thing might still possible in Spirit 2.x...But if so, it's definitely an advanced feature (or just really poorly documented). I have thought of another way to get such an ability into Goldie, and it would be very easy-to-use, but it would also be a fairly non-trivial to implement. And really, I'm starting to question again how important it would *really* be, at least initially. When I think of typical code, usually only a small amount of it is made up of the the sorts of terminals that would need extra processing. I have to admit, I still have no idea whether or not it would be worth it to get Goldie into Phobos. Maybe, maybe not, I dunno. I think popular opinion would probably be the best gauge of that. It seems like we're the only ones still in this thread, though...maybe that's a bad sign? ;) I do still think that if your primary goal is to provide parsing of D code through Phobos, then adapting DDMD would be the best best. Goldie would be more appropriate if customized lexing/parsing is the goal.