Let's stop parser Hell

2012-07-05 Thread Denis Shelomovskij
There are more and more projects requiring parsing D code (IDE plugins, DCT and dfmt now). We definitely need a _standard_ fast D parser (suitable as tokenizer). We already have a good one at least in Visual D. Maybe dmd's parser is faster. If so, it can be (easily?) rewritten in D. We also al

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 12:11:33 UTC, Denis Shelomovskij wrote: There are more and more projects requiring parsing D code (IDE plugins, DCT and dfmt now). We definitely need a _standard_ fast D parser (suitable as tokenizer). We already have a good one at least in Visual D. Maybe dmd's pa

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
Forgot to add DDMD, which also has been forked and redesigned recently by someone (thus two more different compiler frontends). But overall I doubt that any project could become standard very soon.

Re: Let's stop parser Hell

2012-07-05 Thread Dmitry Olshansky
On 05-Jul-12 16:11, Denis Shelomovskij wrote: There are more and more projects requiring parsing D code (IDE plugins, DCT and dfmt now). We definitely need a _standard_ fast D parser (suitable as tokenizer). Then do it. It's all about having something so obviously good, fast and flexible that

Re: Let's stop parser Hell

2012-07-05 Thread Denis Shelomovskij
05.07.2012 16:30, Roman D. Boiko пишет: Forgot to add DDMD, which also has been forked and redesigned recently by someone (thus two more different compiler frontends). But overall I doubt that any project could become standard very soon. Why? Even were they all almost equal we can select any o

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 12:32:19 UTC, Dmitry Olshansky wrote: Then do it. It's all about having something so obviously good, fast and flexible that other stuff can be considered only as toys. I might do one, but I'd rather just help other folks make it faster ;) Would you help to make P

Re: Let's stop parser Hell

2012-07-05 Thread Dmitry Olshansky
On 05-Jul-12 17:04, Roman D. Boiko wrote: On Thursday, 5 July 2012 at 12:32:19 UTC, Dmitry Olshansky wrote: Then do it. It's all about having something so obviously good, fast and flexible that other stuff can be considered only as toys. I might do one, but I'd rather just help other folks make

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 12:53:02 UTC, Denis Shelomovskij wrote: 05.07.2012 16:30, Roman D. Boiko пишет: Forgot to add DDMD, which also has been forked and redesigned recently by someone (thus two more different compiler frontends). But overall I doubt that any project could become standard

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 13:05:41 UTC, Dmitry Olshansky wrote: On 05-Jul-12 17:04, Roman D. Boiko wrote: Well why not. But first I'll need to deliver some stuff on my GSOC project. I bet that after you finish with GSOC optimizing Pegged will not be less relevant than it is now :) And as for

Re: Let's stop parser Hell

2012-07-05 Thread Andrei Alexandrescu
On 7/5/12 9:05 AM, Dmitry Olshansky wrote: On 05-Jul-12 17:04, Roman D. Boiko wrote: On Thursday, 5 July 2012 at 12:32:19 UTC, Dmitry Olshansky wrote: Then do it. It's all about having something so obviously good, fast and flexible that other stuff can be considered only as toys. I might do on

Re: Let's stop parser Hell

2012-07-05 Thread Caligo
Is the actual grammar available somewhere? The online language reference is all we got I guess? DMD doesn't seem to be using yacc/bison either. On Thu, Jul 5, 2012 at 7:11 AM, Denis Shelomovskij wrote: > There are more and more projects requiring parsing D code (IDE plugins, DCT > and dfmt now).

Re: Let's stop parser Hell

2012-07-05 Thread Jacob Carlborg
On 2012-07-05 15:08, Roman D. Boiko wrote: Anyway I propose to enumerate major use cases first. Haven't we already done that a couple of times. -- /Jacob Carlborg

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 15:40:53 UTC, Jacob Carlborg wrote: On 2012-07-05 15:08, Roman D. Boiko wrote: Anyway I propose to enumerate major use cases first. Haven't we already done that a couple of times. Well, we did something like that for DCT... but I doubt that it would fit general

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 15:42:22 UTC, Caligo wrote: Is the actual grammar available somewhere? The online language reference is all we got I guess? DMD doesn't seem to be using yacc/bison either. In PEG format, yes (not necessarily correct): https://github.com/roman-d-boiko/Pegged/blob/dct/

Re: Let's stop parser Hell

2012-07-05 Thread Jacob Carlborg
On 2012-07-05 18:04, Roman D. Boiko wrote: Well, we did something like that for DCT... but I doubt that it would fit general needs. Why wouldn't it. If we had, why haven't they been analyzed, classified, discussed, etc.? Or have they? I don't know. Here is what I wrote for DCT: * IDE inte

Re: Let's stop parser Hell

2012-07-05 Thread Dmitry Olshansky
On 05-Jul-12 18:29, Andrei Alexandrescu wrote: On 7/5/12 9:05 AM, Dmitry Olshansky wrote: On 05-Jul-12 17:04, Roman D. Boiko wrote: On Thursday, 5 July 2012 at 12:32:19 UTC, Dmitry Olshansky wrote: Then do it. It's all about having something so obviously good, fast and flexible that other stuf

Re: Let's stop parser Hell

2012-07-05 Thread Jonathan M Davis
On Thursday, July 05, 2012 18:04:11 Roman D. Boiko wrote: > On Thursday, 5 July 2012 at 15:40:53 UTC, Jacob Carlborg wrote: > > On 2012-07-05 15:08, Roman D. Boiko wrote: > >> Anyway I propose to enumerate major use cases first. > > > > Haven't we already done that a couple of times. > > Well, we

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 16:14:27 UTC, Jacob Carlborg wrote: Original post: http://www.digitalmars.com/d/archives/digitalmars/D/DCT_use_cases_-_draft_168106.html#N168141 OK, fairly complete. Let it be the starting point. Why not put it on some wiki, then add some more, discuss, vote, etc.?

Re: Let's stop parser Hell

2012-07-05 Thread Jacob Carlborg
On 2012-07-05 18:32, Roman D. Boiko wrote: My vote would be for Pegged, I guess. Aren't you voting on your own project :) -- /Jacob Carlborg

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 16:28:57 UTC, Jonathan M Davis wrote: It's all too common for someone to suggest that we should do something or implement something without ever attempting to do it themselves, and in general, stuff around here gets done because someone really wants it done, takes th

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 16:38:27 UTC, Jacob Carlborg wrote: On 2012-07-05 18:32, Roman D. Boiko wrote: My vote would be for Pegged, I guess. Aren't you voting on your own project :) Well, I'm going to base parsing on Pegged, after tweaking it to my needs.

Re: Let's stop parser Hell

2012-07-05 Thread Denis Shelomovskij
05.07.2012 20:14, Jacob Carlborg пишет: On 2012-07-05 18:04, Roman D. Boiko wrote: Well, we did something like that for DCT... but I doubt that it would fit general needs. Why wouldn't it. If we had, why haven't they been analyzed, classified, discussed, etc.? Or have they? I don't know.

Re: Let's stop parser Hell

2012-07-05 Thread Denis Shelomovskij
05.07.2012 20:28, Jonathan M Davis пишет: It's all too common for someone to suggest that we should do something or implement something without ever attempting to do it themselves, and in general, stuff around here gets done because someone really wants it done, takes the time to do it, and sees

Re: Let's stop parser Hell

2012-07-05 Thread Philippe Sigaud
>> On 2012-07-05 18:32, Roman D. Boiko wrote: >> >>> My vote would be for Pegged, I guess. As much as I'm flattered by that, my current impression is Pegged is very far from being performant. As a proof-of-concept that, in D, it's possible to parse a string and create a parse tree at compile-tim

Re: Let's stop parser Hell

2012-07-05 Thread Jonathan M Davis
On Thursday, July 05, 2012 22:23:00 Denis Shelomovskij wrote: > 05.07.2012 20:28, Jonathan M Davis пишет: > > It's all too common for someone to suggest that we should > > do something or implement something without ever attempting to do it > > themselves, and in general, stuff around here gets don

Re: Let's stop parser Hell

2012-07-05 Thread Andrei Alexandrescu
On 7/5/12 12:39 PM, Roman D. Boiko wrote: On Thursday, 5 July 2012 at 16:28:57 UTC, Jonathan M Davis wrote: It's all too common for someone to suggest that we should do something or implement something without ever attempting to do it themselves, and in general, stuff around here gets done becau

Re: Let's stop parser Hell

2012-07-05 Thread Andrei Alexandrescu
On 7/5/12 2:16 PM, Philippe Sigaud wrote: As much as I'm flattered by that, my current impression is Pegged is very far from being performant. As a proof-of-concept that, in D, it's possible to parse a string and create a parse tree at compile-time and then generate code from this, it's also su

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 18:17:06 UTC, Philippe Sigaud wrote: On 2012-07-05 18:32, Roman D. Boiko wrote: My vote would be for Pegged, I guess. As much as I'm flattered by that, my current impression is Pegged is very far from being performant. As a proof-of-concept that, in D, it's pos

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 18:28:21 UTC, Andrei Alexandrescu wrote: On 7/5/12 2:16 PM, Philippe Sigaud wrote: So Pegged or any other generator should *not* get the community focus right now. Pegged should be the focus. +10 (can I vote ten times?) My plan would be as follow: 1- assemble a

Re: Let's stop parser Hell

2012-07-05 Thread Dmitry Olshansky
On 05-Jul-12 22:23, Denis Shelomovskij wrote: 05.07.2012 20:28, Jonathan M Davis пишет: It's all too common for someone to suggest that we should do something or implement something without ever attempting to do it themselves, and in general, stuff around here gets done because someone really wa

Re: Let's stop parser Hell

2012-07-05 Thread Dmitry Olshansky
On 05-Jul-12 22:22, Andrei Alexandrescu wrote: On 7/5/12 12:39 PM, Roman D. Boiko wrote: On Thursday, 5 July 2012 at 16:28:57 UTC, Jonathan M Davis wrote: It's all too common for someone to suggest that we should do something or implement something without ever attempting to do it themselves, a

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 18:33:50 UTC, Dmitry Olshansky wrote: Count me as interested. CTFE needs more correctness & speed though. So to put it blantly - no it's not possible right NOW. BUT it doesn't prevent us from planing and doing a proof of concept. Pegged seems a good starting point ev

Re: Let's stop parser Hell

2012-07-05 Thread Andrei Alexandrescu
On 7/5/12 2:38 PM, Roman D. Boiko wrote: On Thursday, 5 July 2012 at 18:33:50 UTC, Dmitry Olshansky wrote: Count me as interested. CTFE needs more correctness & speed though. So to put it blantly - no it's not possible right NOW. BUT it doesn't prevent us from planing and doing a proof of concep

Re: Let's stop parser Hell

2012-07-05 Thread Andrei Alexandrescu
On 7/5/12 2:33 PM, Dmitry Olshansky wrote: On 05-Jul-12 22:22, Andrei Alexandrescu wrote: I'd really want to create a task force on this, it is of strategic importance to D. In Walter's own words, no new feature is going to push us forward since we're not really using the great goodies we've got

Re: Let's stop parser Hell

2012-07-05 Thread Philippe Sigaud
On Thu, Jul 5, 2012 at 8:28 PM, Andrei Alexandrescu wrote: > I'll be glad to buy for you any book you might feel you need for this. > Again, there are few things more important for D right now than exploiting > its unmatched-by-competition features to great ends. I don't want the lack > of educat

Re: Let's stop parser Hell

2012-07-05 Thread Jacob Carlborg
On 2012-07-05 20:22, Andrei Alexandrescu wrote: I also am actively opposed to a project of just translating D's front-end to D and dropping it into Phobos because it would smother (a) work on generic parser generators, and (b) strong, dependable formalization of D's syntax. I don't see why you

Re: Let's stop parser Hell

2012-07-05 Thread Tobias Pankrath
On Thursday, 5 July 2012 at 18:17:06 UTC, Philippe Sigaud wrote: As a parser proper, Pegged is awful :-) Nothing I'm ashamed of, as I learn by coding. Hey, I just received the Dragon Book (International Edition), If you are interested in parsing, than I wouldn't recommend the dragon book, but

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 19:54:39 UTC, Philippe Sigaud wrote: On Thu, Jul 5, 2012 at 8:28 PM, Andrei Alexandrescu wrote: I'll be glad to buy for you any book you might feel you need for this. Again, there are few things more important for D right now than exploiting its unmatched-by-compet

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
isKeyword_Dummy (baseline): 2738 [microsec] total, 50 [ns / lookup]. This one calculates a sum of all identifier code units. Included for comparison. isKeyword_Dictionary: 4247 [microsec] total, 242 [ns / lookup]. Check whether an identifier is a keyword using AA (dictionary) lookup. isKeyw

Re: Let's stop parser Hell

2012-07-05 Thread Philippe Sigaud
On Thu, Jul 5, 2012 at 10:00 PM, Tobias Pankrath wrote: > On Thursday, 5 July 2012 at 18:17:06 UTC, Philippe Sigaud wrote: >> >> As a parser proper, Pegged is awful :-) Nothing I'm ashamed of, as I learn >> by coding. Hey, I just received the Dragon Book (International Edition), > > > If you are i

Re: Let's stop parser Hell

2012-07-05 Thread Philippe Sigaud
On Thu, Jul 5, 2012 at 10:02 PM, Roman D. Boiko wrote (on children) > I have four, from 1 to 7 years old... Wouldn't call them a problem, though > :))) Better not telling my wife. She's making noises about having a fourth.

Re: Let's stop parser Hell

2012-07-05 Thread Dmitry Olshansky
On 06-Jul-12 00:16, Roman D. Boiko wrote: isKeyword_Dummy (baseline): 2738 [microsec] total, 50 [ns / lookup]. This one calculates a sum of all identifier code units. Included for comparison. isKeyword_Dictionary: 4247 [microsec] total, 242 [ns / lookup]. Check whether an identifier is a keyw

Re: Let's stop parser Hell

2012-07-05 Thread Andrei Alexandrescu
On 7/5/12 3:54 PM, Philippe Sigaud wrote: (Hesitating between 'The Art of the Metaobject Protocol' and 'Compilers, Techniques and Tools', right now) Former sux latter rox. Andrei

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 20:28:32 UTC, Philippe Sigaud wrote: Hmm 72 € by Springer, 55 € on Amazon. Something is not right. Paperback vs perfect bound maybe? http://www.komkon.org/~sher/books/parsing_techniques_2008.pdf Not sure that it is legal, but definitely free.

Re: Let's stop parser Hell

2012-07-05 Thread deadalnix
Le 05/07/2012 18:32, Roman D. Boiko a écrit : On Thursday, 5 July 2012 at 16:14:27 UTC, Jacob Carlborg wrote: Original post: http://www.digitalmars.com/d/archives/digitalmars/D/DCT_use_cases_-_draft_168106.html#N168141 OK, fairly complete. Let it be the starting point. Why not put it on some

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 22:11:41 UTC, deadalnix wrote: Why not program instead of doing bureaucracy ? To avoid programming things which are not needed or don't fit. I've thrown away several implementations already... time to reflect a little :) But, actually, for DCT I do know what I ne

Re: Let's stop parser Hell

2012-07-05 Thread deadalnix
Le 05/07/2012 18:28, Jonathan M Davis a écrit : On Thursday, July 05, 2012 18:04:11 Roman D. Boiko wrote: On Thursday, 5 July 2012 at 15:40:53 UTC, Jacob Carlborg wrote: On 2012-07-05 15:08, Roman D. Boiko wrote: Anyway I propose to enumerate major use cases first. Haven't we already done th

Re: Let's stop parser Hell

2012-07-05 Thread deadalnix
Le 06/07/2012 00:21, Roman D. Boiko a écrit : On Thursday, 5 July 2012 at 22:11:41 UTC, deadalnix wrote: Why not program instead of doing bureaucracy ? To avoid programming things which are not needed or don't fit. I've thrown away several implementations already... time to reflect a little :)

Re: Let's stop parser Hell

2012-07-05 Thread Roman D. Boiko
On Thursday, 5 July 2012 at 22:32:29 UTC, deadalnix wrote: Well you did the mistake to introduce an over complex mechanism in log(n) to get back to source code when an obvious one in O(1) exists. Let me doubt. I'm fine with that, let you doubt.

Re: Let's stop parser Hell

2012-07-05 Thread Jonathan M Davis
On Friday, July 06, 2012 00:24:11 deadalnix wrote: > Le 05/07/2012 18:28, Jonathan M Davis a écrit : > > On Thursday, July 05, 2012 18:04:11 Roman D. Boiko wrote: > >> On Thursday, 5 July 2012 at 15:40:53 UTC, Jacob Carlborg wrote: > >>> On 2012-07-05 15:08, Roman D. Boiko wrote: > Anyway I pr

Re: Let's stop parser Hell

2012-07-05 Thread Nick Sabalausky
On Thu, 5 Jul 2012 21:54:29 +0200 Philippe Sigaud wrote: > > (Hesitating between 'The Art of the Metaobject Protocol' and > 'Compilers, Techniques and Tools', right now) FWIW, I've found this one to be pretty helpful: http://www.pearsonhighered.com/educator/product/Crafting-A-Compiler/978013606

Re: Let's stop parser Hell

2012-07-06 Thread Philippe Sigaud
On Fri, Jul 6, 2012 at 7:26 AM, Nick Sabalausky wrote: > On Thu, 5 Jul 2012 21:54:29 +0200 > Philippe Sigaud wrote: >> >> (Hesitating between 'The Art of the Metaobject Protocol' and >> 'Compilers, Techniques and Tools', right now) > > FWIW, I've found this one to be pretty helpful: > > http://ww

Re: Let's stop parser Hell

2012-07-06 Thread Nick Sabalausky
On Fri, 6 Jul 2012 09:50:26 +0200 Philippe Sigaud wrote: > On Fri, Jul 6, 2012 at 7:26 AM, Nick Sabalausky > wrote: > > On Thu, 5 Jul 2012 21:54:29 +0200 > > Philippe Sigaud wrote: > >> > >> (Hesitating between 'The Art of the Metaobject Protocol' and > >> 'Compilers, Techniques and Tools', rig

Re: Let's stop parser Hell

2012-07-06 Thread David Piepgrass
Resume: everybody is welcome to join effort of translating DMD front end, and improving Pegged. Also I would like to invite those interested in DCT project to help me with it. Right now I'm trying to understand whether it is possible to incorporate Pegged inside without losing anything critic

Re: Let's stop parser Hell

2012-07-06 Thread Jonathan M Davis
On Thursday, July 05, 2012 14:22:35 Andrei Alexandrescu wrote: > I also am actively opposed to a project of just translating D's > front-end to D and dropping it into Phobos because it would smother (a) > work on generic parser generators, and (b) strong, dependable > formalization of D's syntax

Re: Let's stop parser Hell

2012-07-06 Thread H. S. Teoh
On Sat, Jul 07, 2012 at 02:45:35AM +0200, David Piepgrass wrote: [...] > I'm having some trouble following this thread due to the acronyms: > CTFE, DCT, AA. At least I managed to figure out that CTFE is Compile > Time Function Execution. DCT and AA I already know as Discrete Cosine > Transform and

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 01:47:21 UTC, H. S. Teoh wrote: Frankly, I don't know what DCT stands for (anyone?) That's a working (not final) name of my project, a.k.a. The D Compiler Tools. I've used this acronym in previous threads but didn't explain this time. Current state is very far from

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 00:45:36 UTC, David Piepgrass wrote: This work on parsers might be a good place for me to dive in. I have an interest in parsers and familiarity with LL, LALR, PEGs, and even Pratt parsers (fun!), but I am still inexperienced. ... One thing that has always concerned

Re: Let's stop parser Hell

2012-07-07 Thread Dmitry Olshansky
On 07-Jul-12 13:06, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 00:45:36 UTC, David Piepgrass wrote: This work on parsers might be a good place for me to dive in. I have an interest in parsers and familiarity with LL, LALR, PEGs, and even Pratt parsers (fun!), but I am still inexperienced.

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 09:06:57 UTC, Roman D. Boiko wrote: http://stackoverflow.com/questions/11373644/performance-of-parsers-peg-vs-lalr1-or-llk So far it looks like LALR parsers may have lower constant factors than Packrat. The difference could be minimized by paying attention to pars

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 10:05:30 UTC, Dmitry Olshansky wrote: I can tell you that they are not slower then one another in principle. Quality of implementations trumps every theoretical aspect here. LALR is usually fast even if implemented by book but they are hard to optimize futher and qui

Re: Let's stop parser Hell

2012-07-07 Thread Jacob Carlborg
On 2012-07-07 03:12, Jonathan M Davis wrote: Now, the issue of a "strong, dependable formalization of D's syntax" is another thing entirely. Porting dmd's lexer and parser to Phobos would keep the Phobos implementation in line with dmd much more easily and avoid inconsistencies in the language d

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 10:26:39 UTC, Roman D. Boiko wrote: I think that Pegged can be heavily optimized in performance, and there are no fundamental issues which would make it inherently slower than LALR or a hand-written D-specific parser. Hmm... found an interesting article: http://www

Re: Let's stop parser Hell

2012-07-07 Thread Dmitry Olshansky
On 07-Jul-12 15:30, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 10:26:39 UTC, Roman D. Boiko wrote: I think that Pegged can be heavily optimized in performance, and there are no fundamental issues which would make it inherently slower than LALR or a hand-written D-specific parser. Hmm...

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 11:33:18 UTC, Dmitry Olshansky wrote: Yup, LL(*) is my favorite so far. As for debugging and error recovery they are always a problem. It's interesting, that I also wanted to use DFA for non-terminals (similarly to LL(*)), but was considering their usage as pure pe

Re: Let's stop parser Hell

2012-07-07 Thread Manfred Nowak
Andrei Alexandrescu wrote: > (b) strong, dependable formalization of D's syntax. With this premise a pegged parsing system seems to be wrong, because under pegged parsing unintended syntax ambiguities will stay undetected until some possible minor change will suddenly invalidate parts of the

Re: Let's stop parser Hell

2012-07-07 Thread Tobias Pankrath
I can tell you that they are not slower then one another in principle. Quality of implementations trumps every theoretical aspect here. LALR is usually fast even if implemented by book but they are hard to optimize futher and quite restrictive on "semantic extensions". Proper CFG parsers all

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 14:17:54 UTC, Tobias Pankrath wrote: Proper CFG parsers all are liner-time anyway. To be picky here: The languages that can be parsed in linear time are a strict subset of CFGs. Also any PEG defines a language with linear-time parsing. Some non-context-free languag

Re: Let's stop parser Hell

2012-07-07 Thread Chad J
On 07/05/2012 08:28 AM, Roman D. Boiko wrote: Pegged may eventually become standard, if it will be performanceoptimized and a bit more customizable Interesting, I thought you were hand-writing this stuff. I'm a fan of pegged and made some pull requests, one of which was aimed at making it mo

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 15:42:30 UTC, Chad J wrote: On 07/05/2012 08:28 AM, Roman D. Boiko wrote: Pegged may eventually become standard, if it will be performanceoptimized and a bit more customizable Interesting, I thought you were hand-writing this stuff. I'm a fan of pegged and made som

Re: Let's stop parser Hell

2012-07-07 Thread Tobias Pankrath
On Saturday, 7 July 2012 at 15:39:52 UTC, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 14:17:54 UTC, Tobias Pankrath wrote: Proper CFG parsers all are liner-time anyway. To be picky here: The languages that can be parsed in linear time are a strict subset of CFGs. Also any PEG defines a

Re: Let's stop parser Hell

2012-07-07 Thread Philippe Sigaud
On Sat, Jul 7, 2012 at 6:06 PM, Roman D. Boiko wrote: > The most critical was the one you implemented: ability to define a custom > parse tree. I also needed the ability to use immutable structures (almost) > everywhere, while currently they must be mutable. Also it would be great to > avoid UTF

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 16:14:13 UTC, Tobias Pankrath wrote: Interesting, I thought that PEG ⊂ CFG holds. See section 3.4 of http://bford.info/pub/lang/peg.pdf It contains a simple proof that a non-context-free language (a^n) (b^n) (c^n) can be described with PEG syntax. There are infinit

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 16:27:00 UTC, Philippe Sigaud wrote: I added dstrings because 1- at the time (a few months ago), the lists here were awash in UTF-32 discussions and I thought that'd be the way to go anyway 2- other D parsing libraries seemed to go to UTF32 also (CTPG) 3- I wanted t

Re: Let's stop parser Hell

2012-07-07 Thread Philippe Sigaud
On Sat, Jul 7, 2012 at 6:25 PM, Roman D. Boiko wrote: > On Saturday, 7 July 2012 at 16:14:13 UTC, Tobias Pankrath wrote: >> >> Interesting, I thought that PEG ⊂ CFG holds. > > See section 3.4 of http://bford.info/pub/lang/peg.pdf > > It contains a simple proof that a non-context-free language (a^n

Re: Let's stop parser Hell

2012-07-07 Thread deadalnix
On 07/07/2012 12:05, Dmitry Olshansky wrote: On 07-Jul-12 13:06, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 00:45:36 UTC, David Piepgrass wrote: This work on parsers might be a good place for me to dive in. I have an interest in parsers and familiarity with LL, LALR, PEGs, and even Pratt

Re: Let's stop parser Hell

2012-07-07 Thread deadalnix
On 07/07/2012 13:05, Jacob Carlborg wrote: On 2012-07-07 03:12, Jonathan M Davis wrote: Now, the issue of a "strong, dependable formalization of D's syntax" is another thing entirely. Porting dmd's lexer and parser to Phobos would keep the Phobos implementation in line with dmd much more easily

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 16:49:09 UTC, deadalnix wrote: I tried that. This is almost impossible, dmd's parser and AST are very tightly mixed with dmd's internals. +1

Re: Let's stop parser Hell

2012-07-07 Thread Chad J
On 07/07/2012 12:37 PM, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 16:27:00 UTC, Philippe Sigaud wrote: Note that PEG does not impose to use packrat parsing, even though it was developed to use it. I think it's a historical 'accident' that put the two together: Bryan Ford thesis used the

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 16:56:06 UTC, Chad J wrote: Yeah, it's good to hear this notion reinforced. I had this suspicion that the packrat parser is not necessarily the best/fastest solution, mostly because of the large allocation that has to happen before you get O(n) performance. Thus I

Re: Let's stop parser Hell

2012-07-07 Thread Jonathan M Davis
On Saturday, July 07, 2012 18:37:54 Roman D. Boiko wrote: > On Saturday, 7 July 2012 at 16:27:00 UTC, Philippe Sigaud wrote: > > I added dstrings because > > > > 1- at the time (a few months ago), the lists here were awash in > > UTF-32 > > discussions and I thought that'd be the way to go anyway

Re: Let's stop parser Hell

2012-07-07 Thread Jonathan M Davis
On Saturday, July 07, 2012 13:05:29 Jacob Carlborg wrote: > On 2012-07-07 03:12, Jonathan M Davis wrote: > > Now, the issue of a "strong, dependable formalization of D's syntax" is > > another thing entirely. Porting dmd's lexer and parser to Phobos would > > keep > > the Phobos implementation in l

Re: Let's stop parser Hell

2012-07-07 Thread Andrei Alexandrescu
On 7/7/12 6:05 AM, Dmitry Olshansky wrote: I can tell you that they are not slower then one another in principle. Quality of implementations trumps every theoretical aspect here. LALR is usually fast even if implemented by book but they are hard to optimize futher and quite restrictive on "semant

Re: Let's stop parser Hell

2012-07-07 Thread Andrei Alexandrescu
On 7/7/12 6:24 AM, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 09:06:57 UTC, Roman D. Boiko wrote: http://stackoverflow.com/questions/11373644/performance-of-parsers-peg-vs-lalr1-or-llk So far it looks like LALR parsers may have lower constant factors than Packrat. The difference could

Re: Let's stop parser Hell

2012-07-07 Thread David Piepgrass
Note that PEG does not impose to use packrat parsing, even though it was developed to use it. I think it's a historical 'accident' that put the two together: Bryan Ford thesis used the two together. Interesting. After trying to use ANTLR-C# several years back, I got disillusioned because nobo

Re: Let's stop parser Hell

2012-07-07 Thread Andrei Alexandrescu
On 7/7/12 7:33 AM, Dmitry Olshansky wrote: On 07-Jul-12 15:30, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 10:26:39 UTC, Roman D. Boiko wrote: I think that Pegged can be heavily optimized in performance, and there are no fundamental issues which would make it inherently slower than LALR o

Re: Let's stop parser Hell

2012-07-07 Thread Chad J
On 07/07/2012 01:01 PM, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 16:56:06 UTC, Chad J wrote: Yeah, it's good to hear this notion reinforced. I had this suspicion that the packrat parser is not necessarily the best/fastest solution, mostly because of the large allocation that has to happ

Re: Let's stop parser Hell

2012-07-07 Thread Chad J
On 07/07/2012 02:23 PM, David Piepgrass wrote: Note that PEG does not impose to use packrat parsing, even though it was developed to use it. I think it's a historical 'accident' that put the two together: Bryan Ford thesis used the two together. Interesting. After trying to use ANTLR-C# several

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 18:55:57 UTC, Chad J wrote: I was thinking the same thing. My intent is to create a kind of regular-expression-of-nodes with push/pop operators to recognize ascent and descent on the tree. Such a regular expression would allow one to capture subtrees out of genera

Re: Let's stop parser Hell

2012-07-07 Thread Dmitry Olshansky
On 07-Jul-12 22:25, Andrei Alexandrescu wrote: On 7/7/12 7:33 AM, Dmitry Olshansky wrote: On 07-Jul-12 15:30, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 10:26:39 UTC, Roman D. Boiko wrote: I think that Pegged can be heavily optimized in performance, and there are no fundamental issues w

Re: Let's stop parser Hell

2012-07-07 Thread Dmitry Olshansky
On 07-Jul-12 22:23, Andrei Alexandrescu wrote: On 7/7/12 6:24 AM, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 09:06:57 UTC, Roman D. Boiko wrote: http://stackoverflow.com/questions/11373644/performance-of-parsers-peg-vs-lalr1-or-llk So far it looks like LALR parsers may have lower con

Re: Let's stop parser Hell

2012-07-07 Thread Dmitry Olshansky
On 07-Jul-12 22:29, Chad J wrote: On 07/07/2012 01:01 PM, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 16:56:06 UTC, Chad J wrote: Yeah, it's good to hear this notion reinforced. I had this suspicion that the packrat parser is not necessarily the best/fastest solution, mostly because of th

Re: Let's stop parser Hell

2012-07-07 Thread Dmitry Olshansky
On 07-Jul-12 20:56, Chad J wrote: On 07/07/2012 12:37 PM, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 16:27:00 UTC, Philippe Sigaud wrote: Note that PEG does not impose to use packrat parsing, even though it was developed to use it. I think it's a historical 'accident' that put the two t

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 19:29:47 UTC, Dmitry Olshansky wrote: Another idea that I've never realized is to add operator precedence grammar to pegged. Of course it should fit naturally with traditional PEG, for instance taking responsibility for parsing expressions. But that's already avail

Re: Let's stop parser Hell

2012-07-07 Thread Dmitry Olshansky
On 07-Jul-12 23:35, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 19:29:47 UTC, Dmitry Olshansky wrote: Another idea that I've never realized is to add operator precedence grammar to pegged. Of course it should fit naturally with traditional PEG, for instance taking responsibility for parsin

Re: Let's stop parser Hell

2012-07-07 Thread Timon Gehr
On 07/07/2012 09:35 PM, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 19:29:47 UTC, Dmitry Olshansky wrote: Another idea that I've never realized is to add operator precedence grammar to pegged. Of course it should fit naturally with traditional PEG, for instance taking responsibility for pa

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 19:50:37 UTC, Timon Gehr wrote: http://en.wikipedia.org/wiki/Operator-precedence_parser#Precedence_climbing_method OK, at least I didn't misunderstand. So my question was whether the alternative that I described and which exists in PEG is somehow worse than OPP. Wi

Re: Let's stop parser Hell

2012-07-07 Thread Andrei Alexandrescu
On 7/7/12 1:01 PM, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 16:56:06 UTC, Chad J wrote: Yeah, it's good to hear this notion reinforced. I had this suspicion that the packrat parser is not necessarily the best/fastest solution, mostly because of the large allocation that has to happen be

Re: Let's stop parser Hell

2012-07-07 Thread Andrei Alexandrescu
On 7/7/12 3:17 PM, Dmitry Olshansky wrote: I'll have to point out that the whole point about integrated lexing is moot. It's more of liability then benefit. At very least it's just implementation curiosity not advantage. Interesting. I'll have to ask for more details on why. Andrei

Re: Let's stop parser Hell

2012-07-07 Thread Roman D. Boiko
On Saturday, 7 July 2012 at 19:58:37 UTC, Roman D. Boiko wrote: On Saturday, 7 July 2012 at 19:50:37 UTC, Timon Gehr wrote: http://en.wikipedia.org/wiki/Operator-precedence_parser#Precedence_climbing_method OK, at least I didn't misunderstand. So my question was whether the alternative that I

  1   2   3   >