"Andrei Alexandrescu" <seewebsiteforem...@erdani.org> wrote in message news:i9spsa$ll...@digitalmars.com... > On 10/22/10 14:02 CDT, Tomek Sowinski wrote: >> Dnia 22-10-2010 o 00:01:21 Walter Bright <newshou...@digitalmars.com> >> napisal(a): >> >>> As we all know, tool support is important for D's success. Making >>> tools easier to build will help with that. >>> >>> To that end, I think we need a lexer for the standard library - >>> std.lang.d.lex. It would be helpful in writing color syntax >>> highlighting filters, pretty printers, repl, doc generators, static >>> analyzers, and even D compilers. >>> >> >> Interesting idea. Here's another: D will soon need bindings for CORBA, >> Thrift, etc, so lexers will have to be written all over to grok >> interface files. Perhaps a generic tokenizer which can be parametrized >> with a lexical grammar would bring more ROI, I got a hunch D's templates >> are strong enough to pull this off without any source code generation >> ala JavaCC. The books I read on compilers say tokenization is a solved >> problem, so the theory part on what a good abstraction should be is >> done. What you think? > > Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer > generator. >
FWIW, I've been converting my Goldie lexing/parsing library/toolset ( http://www.dsource.org/projects/goldie ) to D2/Phobos, and that should have a release sometime in the next couple months or so. I'm not sure it would really be appropriate for Phobos since it's pretty big, comes with a bunch of tools (one of which is built in C# ATM), it's not range-ified yet, probably doesn't use Phobos coding conventions, and relies on one of my other libraries/tools. But it does do generalized lexing/parsing (LALR) via the GOLD ( http://www.devincook.com/goldparser/ ) grammar file formats, can optionally generate source files for better compile-time checking (for instance, so Token!"<Statemnt>" will generate a compile-time error), has full documentation, and I'm working on a tool/lib that will compile the grammars without having to use the Windows/GUI-based GOLD Parser Builder tool.
