Lexer and parser generators using CTFE

2012-02-28 Thread Andrei Alexandrescu
I'm starting a new thread on this because I think the matter is of strategic importance. We all felt for a long time that there's a lot of potential in CTFE, and potential applications have been discussed more than a few times, ranging from formatting strings parsed to DSLs and parser generato

Re: Lexer and parser generators using CTFE

2012-02-28 Thread CTFE-4-the-win
On Tuesday, 28 February 2012 at 07:59:16 UTC, Andrei Alexandrescu wrote: I'm starting a new thread on this because I think the matter is of strategic importance. We all felt for a long time that there's a lot of potential in CTFE, and potential applications have been discussed more than a few

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Hisayuki Mima
(2012/02/28 16:59), Andrei Alexandrescu wrote: I'm starting a new thread on this because I think the matter is of strategic importance. We all felt for a long time that there's a lot of potential in CTFE, and potential applications have been discussed more than a few times, ranging from formatti

Re: Lexer and parser generators using CTFE

2012-02-28 Thread bls
On 02/28/2012 12:36 AM, CTFE-4-the-win wrote: We need to have a easy-to-use, complete, seamless, and efficient lexer-parser generator combo in Phobos, pronto. The lexer itself could use a character-level PEG or a classic automaton, and emit tokens for consumption by a parser generator. The two sh

Re: Lexer and parser generators using CTFE

2012-02-28 Thread H. S. Teoh
On Tue, Feb 28, 2012 at 01:59:21AM -0600, Andrei Alexandrescu wrote: > I'm starting a new thread on this because I think the matter is of > strategic importance. +1. > We all felt for a long time that there's a lot of potential in CTFE, > and potential applications have been discussed more than

Re: Lexer and parser generators using CTFE

2012-02-28 Thread mist
Something similar to Boost::Spirit2 but with sane syntax and better debugging would have been awesome.

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Martin Nowak
On Tue, 28 Feb 2012 08:59:21 +0100, Andrei Alexandrescu wrote: I'm starting a new thread on this because I think the matter is of strategic importance. We all felt for a long time that there's a lot of potential in CTFE, and potential applications have been discussed more than a few time

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Dmitry Olshansky
On 28.02.2012 22:46, Martin Nowak wrote: On Tue, 28 Feb 2012 08:59:21 +0100, Andrei Alexandrescu wrote: I'm starting a new thread on this because I think the matter is of strategic importance. We all felt for a long time that there's a lot of potential in CTFE, and potential applications have

Re: Lexer and parser generators using CTFE

2012-02-28 Thread H. S. Teoh
On Tue, Feb 28, 2012 at 07:46:04PM +0100, Martin Nowak wrote: [...] > I wrote a generic lexer generator some time ago. > It already let to some compiler O(N^2) optimizations, because the token > declarations sneak into the mangling :(. > I also finally added a workaround for a remaining CTFE bug (#

Re: Lexer and parser generators using CTFE

2012-02-28 Thread H. S. Teoh
On Tue, Feb 28, 2012 at 11:52:52PM +0400, Dmitry Olshansky wrote: > On 28.02.2012 22:46, Martin Nowak wrote: > >On Tue, 28 Feb 2012 08:59:21 +0100, Andrei Alexandrescu > > wrote: [...] > >>We need to have a easy-to-use, complete, seamless, and efficient > >>lexer-parser generator combo in Phobos, p

Re: Lexer and parser generators using CTFE

2012-02-28 Thread deadalnix
Le 28/02/2012 21:02, H. S. Teoh a écrit : OTOH, perhaps once we start writing a lexer/parser generator, that will force us to fix the I/O and allocator system. Without a good project to motivate us to fix these tedious issues, it seems that we just lose inspiration to do it after a while. std.

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Martin Nowak
On Tue, 28 Feb 2012 21:02:24 +0100, H. S. Teoh wrote: On Tue, Feb 28, 2012 at 07:46:04PM +0100, Martin Nowak wrote: [...] I wrote a generic lexer generator some time ago. It already let to some compiler O(N^2) optimizations, because the token declarations sneak into the mangling :(. I also f

Re: Lexer and parser generators using CTFE

2012-02-28 Thread H. S. Teoh
On Tue, Feb 28, 2012 at 10:03:42PM +0100, Martin Nowak wrote: [...] > I won't deny that the combination of CTFE text processing and static > introspection could improve on this. It could be made more feasible by > some conventions, e.g. parse result always uses structs or classes and > built-in arr

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Christopher Bergqvist
On Tuesday, 28 February 2012 at 08:36:13 UTC, CTFE-4-the-win wrote: On Tuesday, 28 February 2012 at 07:59:16 UTC, Andrei Alexandrescu wrote: I'm starting a new thread on this because I think the matter is of strategic importance. We all felt for a long time that there's a lot of potential in

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Jonathan M Davis
On Wednesday, February 29, 2012 02:16:12 Christopher Bergqvist wrote: > I agree that the current direction of D in this area is > impressive. However, I fail to see a killer-feature in > generating a lexer-parser generator at compile-time instead of > run-time. > > A run-time generator would benef

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Piotr Szturmaj
Jonathan M Davis wrote: On Wednesday, February 29, 2012 02:16:12 Christopher Bergqvist wrote: I agree that the current direction of D in this area is impressive. However, I fail to see a killer-feature in generating a lexer-parser generator at compile-time instead of run-time. A run-time genera

Re: Lexer and parser generators using CTFE

2012-02-28 Thread James Miller
On 29 February 2012 14:16, Christopher Bergqvist wrote: > On Tuesday, 28 February 2012 at 08:36:13 UTC, CTFE-4-the-win wrote: >> >> On Tuesday, 28 February 2012 at 07:59:16 UTC, Andrei Alexandrescu >> wrote: >>> >>> I'm starting a new thread on this because I think the matter is of >>> strategic i

Re: Lexer and parser generators using CTFE

2012-02-28 Thread H. S. Teoh
On Tue, Feb 28, 2012 at 08:25:21PM -0500, Jonathan M Davis wrote: > On Wednesday, February 29, 2012 02:16:12 Christopher Bergqvist wrote: > > I agree that the current direction of D in this area is impressive. > > However, I fail to see a killer-feature in generating a lexer-parser > > generator at

Re: Lexer and parser generators using CTFE

2012-02-28 Thread d coder
> > > > On Wednesday, February 29, 2012 02:16:12 Christopher Bergqvist wrote: > > > I agree that the current direction of D in this area is impressive. > > > However, I fail to see a killer-feature in generating a lexer-parser > > > generator at compile-time instead of run-time. > > > > CTFE parsi

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Nick Sabalausky
"H. S. Teoh" wrote in message news:mailman.215.1330472867.24984.digitalmar...@puremagic.com... > On Tue, Feb 28, 2012 at 10:03:42PM +0100, Martin Nowak wrote: > [...] >> I won't deny that the combination of CTFE text processing and static >> introspection could improve on this. It could be made m

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Nick Sabalausky
"Nick Sabalausky" wrote in message news:jikcca$1vq7$1...@digitalmars.com... > "H. S. Teoh" wrote in message > news:mailman.215.1330472867.24984.digitalmar...@puremagic.com... >> On Tue, Feb 28, 2012 at 10:03:42PM +0100, Martin Nowak wrote: >> [...] >>> I won't deny that the combination of CTFE

Re: Lexer and parser generators using CTFE

2012-02-28 Thread bcs
On 02/27/2012 11:59 PM, Andrei Alexandrescu wrote: I'm starting a new thread on this because I think the matter is of strategic importance. We all felt for a long time that there's a lot of potential in CTFE, and potential applications have been discussed more than a few times, ranging from form

Re: Lexer and parser generators using CTFE

2012-02-28 Thread bcs
On 02/28/2012 08:23 AM, mist wrote: Something similar to Boost::Spirit2 but with sane syntax and better debugging would have been awesome. How about not having to encode the syntax as arithmetic expressions? D can do string parsing of eBNF at compile time if you want that.

Re: Lexer and parser generators using CTFE

2012-02-28 Thread Nick Sabalausky
"Nick Sabalausky" wrote in message news:jikcit$201o$1...@digitalmars.com... > > Hmm, maybe I need to think about what it would take to make Goldie able to > parse at compile-time... > Just gave it a quick shot. It was looking like it might not be too bad, but then I hit: Assertion failure: 'c

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Philippe Sigaud
>> > On Wednesday, February 29, 2012 02:16:12 Christopher Bergqvist wrote: >> > > I agree that the current direction of D in this area is impressive. >> > > However, I fail to see a killer-feature in generating a lexer-parser >> > > generator at compile-time instead of run-time. >> > > > > > CTFE p

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Philippe Sigaud
Le 29 févr. 2012 07:20, "Nick Sabalausky" a écrit : > > "Nick Sabalausky" wrote in message > news:jikcit$201o$1...@digitalmars.com... > > > > Hmm, maybe I need to think about what it would take to make Goldie able to > > parse at compile-time... > > > > Just gave it a quick shot. It was looking l

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Dmitry Olshansky
On 29.02.2012 0:19, H. S. Teoh wrote: On Tue, Feb 28, 2012 at 11:52:52PM +0400, Dmitry Olshansky wrote: On 28.02.2012 22:46, Martin Nowak wrote: On Tue, 28 Feb 2012 08:59:21 +0100, Andrei Alexandrescu wrote: [...] We need to have a easy-to-use, complete, seamless, and efficient lexer-parser

Re: Lexer and parser generators using CTFE

2012-02-29 Thread d coder
> > I'm afraid being in holidays right now means I do not have easy access to > GitHub (no git on a pad, and the computer I use to code right now does not > have any network connection). I'll put all this online in a few days, > because that must seems like the ramblings of a madman right now... >

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Hisayuki Mima
(2012/02/28 23:58), Hisayuki Mima wrote: (2012/02/28 16:59), Andrei Alexandrescu wrote: I'm starting a new thread on this because I think the matter is of strategic importance. We all felt for a long time that there's a lot of potential in CTFE, and potential applications have been discussed mo

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/28/12 8:58 AM, Hisayuki Mima wrote: Certainly, I don't write yet the documentation of my library, ctpg. (But a few examples available here: https://github.com/youkei/ctpg/tree/master/example) Nice! I have a few comments. I should say that they entail a fair amount of work. The current s

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/28/12 9:57 AM, H. S. Teoh wrote: On Tue, Feb 28, 2012 at 01:59:21AM -0600, Andrei Alexandrescu wrote: This is the kind of stuff I've had an eye on for the longest time. I'm saying it's of strategic importance because CTFE technology, though not new and already available with some languages,

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/28/12 10:23 AM, mist wrote: Something similar to Boost::Spirit2 but with sane syntax and better debugging would have been awesome. Exactly. Boost::Spirit2 is a perfect illustration of a dancing bear - using inadequate technology for a specific purpose. Andrei

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/28/12 12:46 PM, Martin Nowak wrote: I wrote a generic lexer generator some time ago. It already let to some compiler O(N^2) optimizations, because the token declarations sneak into the mangling :(. I also finally added a workaround for a remaining CTFE bug (#6815). https://gist.github.com/1

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/28/12 1:52 PM, Dmitry Olshansky wrote: To be truly successful it should have some distinct properties like: - being faster or identical to handwritten stuff we already have (like e.g. std.json ), reminds us of recent INI parser proposal, should be an easy pick for general parser. Yes. -

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Timon Gehr
On 02/28/2012 07:46 PM, Martin Nowak wrote: https://gist.github.com/1255439 - lexer generator https://gist.github.com/1262321 - complete and fast D lexer Well, it is slower at lexing than DMD at parsing. What is the bottleneck?

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/28/12 11:15 PM, Nick Sabalausky wrote: In Goldie, I've taken an inverted approach, which IMHO is easier to use: The types are automatically generated from the grammar, not the other way around. So applying that approach to the above code, it'd be more like this: mixin genGrammar!("myGrammar

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/29/12 12:07 AM, bcs wrote: On 02/28/2012 08:23 AM, mist wrote: Something similar to Boost::Spirit2 but with sane syntax and better debugging would have been awesome. How about not having to encode the syntax as arithmetic expressions? D can do string parsing of eBNF at compile time if you

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/28/12 7:16 PM, Christopher Bergqvist wrote: I agree that the current direction of D in this area is impressive. However, I fail to see a killer-feature in generating a lexer-parser generator at compile-time instead of run-time. A run-time generator would benefit from not having to execute w

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/29/12 3:45 AM, Philippe Sigaud wrote: mixin(Grammar!("Doc <- Node*" "Node <- OpeningTag (Text / Node)* ClosingTag", NodeAction, "OpeningTag <- '<' Identifier '>'", OpeningAction, "ClosingTag <- `'", ClosingAction, "Text <- (!(OpeningTag / ClosingTag) _)+")); That looks about right, but st

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/29/12 3:48 AM, Dmitry Olshansky wrote: Someway of stacking grammars is bonus points for the project. As Andrei started with lexer+parser, I see no problem with it being lexer+parser(+parser)*. What does stacking grammars mean? Anyhow, one thing that would become important is tree walkers

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/29/12 10:03 AM, Hisayuki Mima wrote: A minimum of documentation is now available here: https://github.com/youkei/ctpg/wiki/Home-en Great start, you may want to aggressively add examples. Then people would love using ctpg and will write more documentation, tutorials etc. Andrei

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Nick Sabalausky
"Philippe Sigaud" wrote in message news:mailman.229.1330508922.24984.digitalmar...@puremagic.com... Le 29 févr. 2012 07:20, "Nick Sabalausky" a écrit : > >> >> Just gave it a quick shot. It was looking like it might not be too bad, >but >> then I hit: >> >> Assertion failure: 'ctfeStack.stackPoi

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Nick Sabalausky
"Andrei Alexandrescu" wrote in message news:jilkj6$1a50$2...@digitalmars.com... > On 2/28/12 11:15 PM, Nick Sabalausky wrote: >> In Goldie, I've taken an inverted approach, which IMHO is easier to use: >> The >> types are automatically generated from the grammar, not the other way >> around. So

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/29/12 11:05 AM, Nick Sabalausky wrote: What? We agree? ;) It's 2012 after all. Andrei

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Nick Sabalausky
"Andrei Alexandrescu" wrote in message news:jiljf7$18em$1...@digitalmars.com... > On 2/28/12 9:57 AM, H. S. Teoh wrote: >> On Tue, Feb 28, 2012 at 01:59:21AM -0600, Andrei Alexandrescu wrote: >>> This is the kind of stuff I've had an eye on for the longest time. >>> I'm saying it's of strategic i

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Nick Sabalausky
"Andrei Alexandrescu" wrote in message news:jiljsg$193s$1...@digitalmars.com... > On 2/28/12 1:52 PM, Dmitry Olshansky wrote: >> - have reasonable compile times and memory consumption (though it will >> only improve over time) > > Yes. I guess PEGs have problems there. > Probably LR, too, unless

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Nick Sabalausky
"Nick Sabalausky" wrote in message news:jilmhr$1dul$1...@digitalmars.com... > "Andrei Alexandrescu" wrote in message > news:jiljsg$193s$1...@digitalmars.com... >> On 2/28/12 1:52 PM, Dmitry Olshansky wrote: >>> - have reasonable compile times and memory consumption (though it will >>> only impr

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrei Alexandrescu
On 2/29/12 11:15 AM, Nick Sabalausky wrote: "Andrei Alexandrescu" wrote in message news:jiljsg$193s$1...@digitalmars.com... On 2/28/12 1:52 PM, Dmitry Olshansky wrote: - have reasonable compile times and memory consumption (though it will only improve over time) Yes. I guess PEGs have proble

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Nick Sabalausky
"Andrei Alexandrescu" wrote in message news:jilnjg$1eu9$3...@digitalmars.com... > On 2/29/12 11:15 AM, Nick Sabalausky wrote: >> "Andrei Alexandrescu" wrote in message >> news:jiljsg$193s$1...@digitalmars.com... >>> On 2/28/12 1:52 PM, Dmitry Olshansky wrote: - have reasonable compile times

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Martin Nowak
On Wed, 29 Feb 2012 17:41:19 +0100, Timon Gehr wrote: On 02/28/2012 07:46 PM, Martin Nowak wrote: https://gist.github.com/1255439 - lexer generator https://gist.github.com/1262321 - complete and fast D lexer Well, it is slower at lexing than DMD at parsing. What is the bottleneck? No, it

Re: Lexer and parser generators using CTFE

2012-02-29 Thread H. S. Teoh
On Wed, Feb 29, 2012 at 07:28:48PM +0100, Martin Nowak wrote: > On Wed, 29 Feb 2012 17:41:19 +0100, Timon Gehr wrote: > > >On 02/28/2012 07:46 PM, Martin Nowak wrote: > >> > >>https://gist.github.com/1255439 - lexer generator > >>https://gist.github.com/1262321 - complete and fast D lexer > >> >

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Dmitry Olshansky
On 29.02.2012 20:31, Andrei Alexandrescu wrote: Recalling EBNF parser idea that I run with before finally being dragged down by real life. Roughly I thought to follow hybrid LL(*) aproach, while I had a solid plan on doing DFA for lexer and parser lookahead, the other things were more or less f

Re: Lexer and parser generators using CTFE

2012-02-29 Thread d coder
> > > This sounds like something I ran into in my D lexer project. Lexing > Phobos took upwards of 10-15 seconds, which is extremely slow > considering that dmd can *compile* it in just a few seconds. So I did > some profiling, and found out that most of the time was spent formatting > tokens into

Re: Lexer and parser generators using CTFE

2012-02-29 Thread H. S. Teoh
On Thu, Mar 01, 2012 at 12:19:30AM +0530, d coder wrote: > > > > > > This sounds like something I ran into in my D lexer project. Lexing > > Phobos took upwards of 10-15 seconds, which is extremely slow > > considering that dmd can *compile* it in just a few seconds. So I did > > some profiling, an

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Nick Sabalausky
"d coder" wrote in message news:mailman.241.1330541814.24984.digitalmar...@puremagic.com... > > >> >> This sounds like something I ran into in my D lexer project. Lexing >> Phobos took upwards of 10-15 seconds, which is extremely slow >> considering that dmd can *compile* it in just a few seconds

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Timon Gehr
On 02/29/2012 07:28 PM, Martin Nowak wrote: On Wed, 29 Feb 2012 17:41:19 +0100, Timon Gehr wrote: On 02/28/2012 07:46 PM, Martin Nowak wrote: https://gist.github.com/1255439 - lexer generator https://gist.github.com/1262321 - complete and fast D lexer Well, it is slower at lexing than DMD

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Dmitry Olshansky
On 29.02.2012 20:47, Andrei Alexandrescu wrote: On 2/29/12 3:48 AM, Dmitry Olshansky wrote: Someway of stacking grammars is bonus points for the project. As Andrei started with lexer+parser, I see no problem with it being lexer+parser(+parser)*. What does stacking grammars mean? In general

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Martin Nowak
On Wed, 29 Feb 2012 20:30:43 +0100, Timon Gehr wrote: On 02/29/2012 07:28 PM, Martin Nowak wrote: On Wed, 29 Feb 2012 17:41:19 +0100, Timon Gehr wrote: On 02/28/2012 07:46 PM, Martin Nowak wrote: https://gist.github.com/1255439 - lexer generator https://gist.github.com/1262321 - complet

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Timon Gehr
On 02/29/2012 09:03 PM, Martin Nowak wrote: On Wed, 29 Feb 2012 20:30:43 +0100, Timon Gehr wrote: On 02/29/2012 07:28 PM, Martin Nowak wrote: On Wed, 29 Feb 2012 17:41:19 +0100, Timon Gehr wrote: On 02/28/2012 07:46 PM, Martin Nowak wrote: https://gist.github.com/1255439 - lexer generato

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Christopher Bergqvist
On Wednesday, 29 February 2012 at 16:41:22 UTC, Andrei Alexandrescu wrote: On 2/28/12 7:16 PM, Christopher Bergqvist wrote: What am I failing to pick up on? Barrier of entry and granularity of approach, I think. Currently if one wants to parse some simple grammar, there are options such as (

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Martin Nowak
On Wed, 29 Feb 2012 21:30:57 +0100, Timon Gehr wrote: On 02/29/2012 09:03 PM, Martin Nowak wrote: On Wed, 29 Feb 2012 20:30:43 +0100, Timon Gehr wrote: On 02/29/2012 07:28 PM, Martin Nowak wrote: On Wed, 29 Feb 2012 17:41:19 +0100, Timon Gehr wrote: On 02/28/2012 07:46 PM, Martin Nowa

Re: Lexer and parser generators using CTFE

2012-02-29 Thread H. S. Teoh
On Thu, Mar 01, 2012 at 12:04:39AM +0100, Martin Nowak wrote: [...] > Mmh, I've retested and you're right dmd's lexer is about 2x faster. > The main overhead stems from using ranges and enforce. > > Quick profiling shows that 25% is spent in popFront and > std.utf.stride. Last time I worked on th

Re: Lexer and parser generators using CTFE

2012-02-29 Thread H. S. Teoh
On Wed, Feb 29, 2012 at 10:41:22AM -0600, Andrei Alexandrescu wrote: [...] > An efficient, integrated parser generator would lower the barrier of > entry dramatically - if we play our cards right, even a sprintf > specifier string could be parsed simpler and faster using an embedded > grammar, inst

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Andrej Mitrovic
Why do you need grammar rules for json? Just use a custom-defined struct and pass it to a toJson function like ae's json module. :)

Re: Lexer and parser generators using CTFE

2012-02-29 Thread Marco Leise
Am 29.02.2012, 02:30 Uhr, schrieb Piotr Szturmaj : CTFE code can be much slower than native one, and I would like to see some kind of compiler cache for such code. I second this. As a fan of clean optimizations this is one of the things I tossed around my head for a while. It could use a cach

Re: Lexer and parser generators using CTFE

2012-03-01 Thread deadalnix
Le 29/02/2012 17:45, Andrei Alexandrescu a écrit : On 2/29/12 3:45 AM, Philippe Sigaud wrote: mixin(Grammar!("Doc <- Node*" "Node <- OpeningTag (Text / Node)* ClosingTag", NodeAction, "OpeningTag <- '<' Identifier '>'", OpeningAction, "ClosingTag <- `'", ClosingAction, "Text <- (!(OpeningTag / C

Re: Lexer and parser generators using CTFE

2012-03-01 Thread Philippe Sigaud
>> mixin(Grammar!("Doc <- Node*" >> "Node <- OpeningTag (Text / Node)* ClosingTag", NodeAction, >> "OpeningTag <- '<' Identifier '>'", OpeningAction, >> "ClosingTag <- `'", ClosingAction, >> "Text <- (!(OpeningTag / ClosingTag) _)+")); > > > That looks about right, but still has a fair amount of n

Re: Lexer and parser generators using CTFE

2012-03-01 Thread H. S. Teoh
On Thu, Mar 01, 2012 at 04:10:26PM +0100, Philippe Sigaud wrote: > >> mixin(Grammar!("Doc <- Node*" > >> "Node <- OpeningTag (Text / Node)* ClosingTag", NodeAction, > >> "OpeningTag <- '<' Identifier '>'", OpeningAction, > >> "ClosingTag <- `'", ClosingAction, > >> "Text <- (!(OpeningTag / Closing

Re: Lexer and parser generators using CTFE

2012-03-01 Thread Philippe Sigaud
> > But how to associate actions with a rule, in that case? I mean, some > > rules will have actions, some not. > > You could maybe just put D code in the grammar string, which gets > compiled as a string mixin by CTFE? I could, but one of my driving goals while beginning this project a month ao w

Re: Lexer and parser generators using CTFE

2012-03-01 Thread Andrei Alexandrescu
On 3/1/12 1:54 AM, Marco Leise wrote: Am 29.02.2012, 02:30 Uhr, schrieb Piotr Szturmaj : CTFE code can be much slower than native one, and I would like to see some kind of compiler cache for such code. I second this. As a fan of clean optimizations this is one of the things I tossed around my

Re: Lexer and parser generators using CTFE

2012-03-01 Thread Jacob Carlborg
On 2012-03-01 18:21, Philippe Sigaud wrote: > > But how to associate actions with a rule, in that case? I mean, some > > rules will have actions, some not. > > You could maybe just put D code in the grammar string, which gets > compiled as a string mixin by CTFE? I could, but one of my driv

Re: Lexer and parser generators using CTFE

2012-03-04 Thread Hisayuki Mima
(2012/03/01 1:19), Andrei Alexandrescu wrote: On 2/28/12 8:58 AM, Hisayuki Mima wrote: Certainly, I don't write yet the documentation of my library, ctpg. (But a few examples available here: https://github.com/youkei/ctpg/tree/master/example) Nice! I have a few comments. I should say that they

Re: Lexer and parser generators using CTFE

2012-03-04 Thread Andrei Alexandrescu
On 3/4/12 8:34 AM, Hisayuki Mima wrote: Certainly, "built AST" option is very interesting. I don't know whether building AST is a common case or not because I'm unfamiliar with syntax analysis. But I'd like to complete ctpg as D version of boost::spirit::qi first. Currently, ctpg can be used prob

Re: Lexer and parser generators using CTFE

2012-03-05 Thread Ben Hanson
Hi Nick, On Wednesday, 29 February 2012 at 17:16:44 UTC, Nick Sabalausky wrote: One thing I've been wanting to see in a lexer's regex would be a way to match things like D's nested comments or: I have already implemented a lexer generator that can handle recursion in the token definitions (us

Re: Lexer and parser generators using CTFE

2012-03-05 Thread Ben Hanson
On Sunday, 4 March 2012 at 21:00:00 UTC, Andrei Alexandrescu wrote: You can get away without an AST for the simplest grammars (e.g. printf etc.) but in that case you compete with hand-written code. I wouldn't think of parsing a serious grammar with more than a few productions without generating

Re: Lexer and parser generators using CTFE

2012-03-07 Thread Nick Sabalausky
"Nick Sabalausky" wrote in message news:jikfpi$25cc$1...@digitalmars.com... > "Nick Sabalausky" wrote in message > news:jikcit$201o$1...@digitalmars.com... >> >> Hmm, maybe I need to think about what it would take to make Goldie able >> to parse at compile-time... >> > > Just gave it a quick s

Re: Lexer and parser generators using CTFE

2012-05-27 Thread d coder
> > > Generally a parser generated by other tool and accepting tokens returns > the abstract syntax tree, but it return the evaluated value in the example. > In other words, it does lexical analysis, parsing and (type) converting at > a time. > If you want simply abstract syntax tree, it may be a l

Re: Lexer and parser generators using CTFE

2012-05-27 Thread d coder
> > I would like the parser to effect some side effects. For this purpose, I > tried including the parser mixin into a class, but I got a strange error > saying: > > Error: need 'this' to access member parse > > Ok. I see my folly. At compile time, there would not be any "this" since the class has

Re: Lexer and parser generators using CTFE

2012-05-27 Thread F i L
I'm not sure I follow all the details of what Andrei's suggesting and what's being talked about here, this parser/lexer stuff is still very new to me, so this may be a bit off-topic. However, I thought I'd weigh in on something I was very impressed with about the Nimrod language's direct AST ac

Re: Lexer and parser generators using CTFE

2012-05-27 Thread John Belmonte
On Thursday, 1 March 2012 at 15:10:36 UTC, Philippe Sigaud wrote: mixin(Grammar!("Doc <- Node*" "Node <- OpeningTag (Text / Node)* ClosingTag", NodeAction, "OpeningTag <- '<' Identifier '>'", OpeningAction, "ClosingTag <- `'", ClosingAction, "Text <- (!(OpeningTag / ClosingTag) _)+")); That l

Re: Lexer and parser generators using CTFE

2012-05-28 Thread Jacob Carlborg
On 2012-05-27 22:15, F i L wrote: I'm not sure I follow all the details of what Andrei's suggesting and what's being talked about here, this parser/lexer stuff is still very new to me, so this may be a bit off-topic. However, I thought I'd weigh in on something I was very impressed with about the

Re: Lexer and parser generators using CTFE

2012-05-28 Thread Philippe Sigaud
On Sun, May 27, 2012 at 11:13 PM, John Belmonte wrote: > I'm wondering if people have seen LPeg.  It's a Lua library, but the design > is interesting in that patterns are first class objects which can be > composed with operator overloading. > > http://www.inf.puc-rio.br/~roberto/lpeg/ This is ex

Re: Lexer and parser generators using CTFE

2012-05-28 Thread John Belmonte
On Monday, 28 May 2012 at 12:27:09 UTC, Philippe Sigaud wrote: I played with this idea with my own Pegged (https://github.com/PhilippeSigaud/Pegged), but I wasn't quite convinced by the result, exactly for the reason above. Also, when looking at real-world Spirit examples, I was a bit disappoint

Re: Lexer and parser generators using CTFE

2012-05-29 Thread Philippe Sigaud
On Mon, May 28, 2012 at 11:42 PM, John Belmonte wrote: > Fair enough.  I did notice the following in the markdown PEG though which > could benefit from first class patterns: > > > # Parsers for different kinds of block-level HTML content. > # This is repetitive due to constraints of PEG grammar. >

Re: Lexer and parser generators using CTFE

2012-05-31 Thread Hisayuki Mima
(2012年05月28日 02:31), d coder wrote: Generally a parser generated by other tool and accepting tokens returns the abstract syntax tree, but it return the evaluated value in the example. In other words, it does lexical analysis, parsing and (type) converting at a time. If yo

Re: Lexer and parser generators using CTFE

2012-06-01 Thread Ken
On Tuesday, 28 February 2012 at 07:59:16 UTC, Andrei Alexandrescu wrote: I'm starting a new thread on this because I think the matter is of strategic importance. We all felt for a long time that there's a lot of potential in CTFE, and potential applications have been discussed more than a few

Re: Lexer and parser generators using CTFE

2012-06-01 Thread Andrei Alexandrescu
On 6/1/12 8:39 AM, Ken wrote: Great! So what's the next step? Do we wait for the maintainers of one of the CTFE parser gen packages to drop it in the Phobos Review Queue? Do a reimplementation for Phobos? We could attack this in pieces. Start with a lexer/FSA generator (like Ragel but using CTFE