Re: Formal review of std.lexer

2014-05-05 Thread Dmitry Olshansky via Digitalmars-d
17-Mar-2014 02:13, Martin Nowak пишет: On 02/22/2014 09:31 PM, "Marc Schütz" " wrote: But that still doesn't explain why a custom hash table implementation is necessary. Maybe a lightweight wrapper around built-in AAs is sufficient? I'm also wondering what benefit this hash table provides. G

Re: Formal review of std.lexer

2014-04-15 Thread Alix Pexton via Digitalmars-d
On 14/04/2014 10:34 PM, Brian Schott wrote: ubyte[] is required for the lexer to work quickly. That lexer range is designed to keep track of the column and line numbers. I can understand that speed requires the input to be handled as bytes instead of chars, but the restriction to an ubyte[] ov

Re: Formal review of std.lexer

2014-04-14 Thread Brian Schott
On Monday, 14 April 2014 at 20:43:58 UTC, Alix Pexton wrote: I know the official review period is long past but I'd not had a good look at this module until this past weekend. Last year I had been working on my own xml lexer/parser but as of yet I have nothing to show for it so I took a look a

Re: Formal review of std.lexer

2014-04-14 Thread Brian Schott
On Sunday, 16 March 2014 at 22:13:16 UTC, Martin Nowak wrote: On 02/22/2014 09:31 PM, "Marc Schütz" " wrote: But that still doesn't explain why a custom hash table implementation is necessary. Maybe a lightweight wrapper around built-in AAs is sufficient? I'm also wondering what benefit this

Re: Formal review of std.lexer

2014-04-14 Thread Alix Pexton
On 21/02/2014 12:12 PM, Dicebot wrote: http://wiki.dlang.org/Review/std.lexer This is follow-up by Brian to his earlier proposal (http://wiki.dlang.org/Review/std.d.lexer). This time proposed module focuses instead on generic lexer generation as discussed in matching voting thread. Docs: http:/

Re: Formal review of std.lexer

2014-03-16 Thread Martin Nowak
On 02/22/2014 09:31 PM, "Marc Schütz" " wrote: But that still doesn't explain why a custom hash table implementation is necessary. Maybe a lightweight wrapper around built-in AAs is sufficient? I'm also wondering what benefit this hash table provides.

Re: Formal review of std.lexer

2014-03-16 Thread Dicebot
Initial review has finished. Voting will be delayed because Brian is currently busy and there is ongoing Walter's scopebuffer proposal to be processed (per agreement with both Brian and Walter). Anyone late for review can still leave comments, I am sure Brian will take them into consideration

Re: Formal review of std.lexer

2014-03-10 Thread Dicebot
On Wednesday, 26 February 2014 at 18:07:37 UTC, Jacob Carlborg wrote: On 2014-02-26 00:25, Dicebot wrote: Don't know if it makes sense to introduce random package categorizatin. I'd love to see more hierarchy in Phobos too but we'd first need to agree to package separation principles then.

Re: Formal review of std.lexer

2014-03-10 Thread Dicebot
Reminder about benchmarks. By the way, is generated lexer usable at CTFE? Imaginary use case : easier DSL implementation.

Re: Formal review of std.lexer

2014-03-03 Thread Dicebot
Bringing this back to the front page.

Re: Formal review of std.lexer

2014-02-26 Thread Jacob Carlborg
On 2014-02-26 00:25, Dicebot wrote: Don't know if it makes sense to introduce random package categorizatin. I'd love to see more hierarchy in Phobos too but we'd first need to agree to package separation principles then. Then that's what we need to do. I don't want any more top level modules.

Re: Formal review of std.lexer

2014-02-26 Thread Jacob Carlborg
On 2014-02-26 16:18, Andrei Alexandrescu wrote: I think we wouldn't want to add one more package for each language supported. That's exactly what we want, preferably in a common package: std.language.d.lexer std.language.d.ast std.language.d.parser std.language.xml.lexer std.language.xml.par

Re: Formal review of std.lexer

2014-02-26 Thread Jacob Carlborg
On 2014-02-26 07:58, Brian Schott wrote: I think that's a bit backwards. I'd rather have std.d.lexer std.d.ast std.d.parser than std.lexer.d std.parser.d std.ast.d I agree with Brian. Although I would have a common package for all languages: std.language.d.lexer std.language.d.ast std.la

Re: Formal review of std.lexer

2014-02-26 Thread Andrei Alexandrescu
On 2/25/14, 10:58 PM, Brian Schott wrote: On Tuesday, 25 February 2014 at 23:17:56 UTC, Meta wrote: std.lexer could be the umbrella for a bunch of different lexers. Then we could have std.lexer.xml, std.lexer.json, etc. I think that's a bit backwards. I'd rather have std.d.lexer std.d.ast std

Re: Formal review of std.lexer

2014-02-26 Thread Dicebot
On Wednesday, 26 February 2014 at 07:00:33 UTC, Brian Schott wrote: On Monday, 24 February 2014 at 19:05:35 UTC, Dicebot wrote: Brian, do you have benchmarks for this proposal similar to ones you have provided in old review threads? (vs DMD frontend lexer) Not yet, but I can create them befor

Re: Formal review of std.lexer

2014-02-25 Thread Brian Schott
On Tuesday, 25 February 2014 at 23:17:56 UTC, Meta wrote: std.lexer could be the umbrella for a bunch of different lexers. Then we could have std.lexer.xml, std.lexer.json, etc. I think that's a bit backwards. I'd rather have std.d.lexer std.d.ast std.d.parser than std.lexer.d std.parser.d s

Re: Formal review of std.lexer

2014-02-25 Thread Brian Schott
On Monday, 24 February 2014 at 19:05:35 UTC, Dicebot wrote: Brian, do you have benchmarks for this proposal similar to ones you have provided in old review threads? (vs DMD frontend lexer) Not yet, but I can create them before the review period ends.

Re: Formal review of std.lexer

2014-02-25 Thread Brian Schott
On Tuesday, 25 February 2014 at 20:21:17 UTC, Andrei Alexandrescu wrote: I think it would be great if we rigged things such that std.lexer simplifies flint and makes it faster. From what I saw Brian has a very similar approach. I'm hoping for a few improvements, which I'll share soon. Andrei

Re: Formal review of std.lexer

2014-02-25 Thread Dicebot
On Tuesday, 25 February 2014 at 20:48:08 UTC, Jacob Carlborg wrote: I don't like it being a new top level module. I would name it std.language.lexer or std.lang.lexer. Don't know if it makes sense to introduce random package categorizatin. I'd love to see more hierarchy in Phobos too but we'd

Re: Formal review of std.lexer

2014-02-25 Thread Meta
On Tuesday, 25 February 2014 at 20:48:08 UTC, Jacob Carlborg wrote: On 2014-02-21 13:12, Dicebot wrote: http://wiki.dlang.org/Review/std.lexer This is follow-up by Brian to his earlier proposal (http://wiki.dlang.org/Review/std.d.lexer). This time proposed module focuses instead on generic lex

Re: Formal review of std.lexer

2014-02-25 Thread Jacob Carlborg
On 2014-02-21 13:12, Dicebot wrote: http://wiki.dlang.org/Review/std.lexer This is follow-up by Brian to his earlier proposal (http://wiki.dlang.org/Review/std.d.lexer). This time proposed module focuses instead on generic lexer generation as discussed in matching voting thread. I don't like i

Re: Formal review of std.lexer

2014-02-25 Thread Andrei Alexandrescu
On 2/24/14, 7:18 PM, Adam Wilson wrote: Note that I as badly as I want std.lexer to be included I want it to pass a rigorous review. This review (and, is passing, subsequent inclusion) has opened up an opportunity to start using D at work that I did not expect and so I am kind of excited about it

Re: Formal review of std.lexer

2014-02-25 Thread Dicebot
On Tuesday, 25 February 2014 at 03:52:01 UTC, H. S. Teoh wrote: Sadly, I haven't had the time to review the proposed code as closely as I'd like. Don't hesitate to ask for extending review period if you need so. Making good review is more important than meeting some arbitrary deadline.

Re: Formal review of std.lexer

2014-02-24 Thread H. S. Teoh
On Mon, Feb 24, 2014 at 07:18:58PM -0800, Adam Wilson wrote: [...] > Note that I as badly as I want std.lexer to be included I want it to > pass a rigorous review. This review (and, is passing, subsequent > inclusion) has opened up an opportunity to start using D at work > that I did not expect and

Re: Formal review of std.lexer

2014-02-24 Thread Adam Wilson
On Mon, 24 Feb 2014 17:22:32 -0800, Meta wrote: On Tuesday, 25 February 2014 at 00:28:26 UTC, Adam Wilson wrote: [SNIP] You're throwing what I said way out of proportion. I was replying to the statement: "No criticism should stop this module being accepted, as we do not have any other le

Re: Formal review of std.lexer

2014-02-24 Thread Meta
On Tuesday, 25 February 2014 at 00:28:26 UTC, Adam Wilson wrote: [SNIP] You're throwing what I said way out of proportion. I was replying to the statement: "No criticism should stop this module being accepted, as we do not have any other lexer in the runtime anyway. Therefore I suggest we ac

Re: Formal review of std.lexer

2014-02-24 Thread Adam Wilson
On Mon, 24 Feb 2014 15:36:43 -0800, Meta wrote: On Monday, 24 February 2014 at 23:07:07 UTC, Adam Wilson wrote: Well, we keep voting down replacement candidates, which incidentally, is exactly what happened with the std.signals replacement, so I view this as an orthogonal issue to whether o

Re: Formal review of std.lexer

2014-02-24 Thread Meta
On Monday, 24 February 2014 at 23:07:07 UTC, Adam Wilson wrote: Well, we keep voting down replacement candidates, which incidentally, is exactly what happened with the std.signals replacement, so I view this as an orthogonal issue to whether or not it should be included after passing a review.

Re: Formal review of std.lexer

2014-02-24 Thread Adam Wilson
On Mon, 24 Feb 2014 14:32:53 -0800, Meta wrote: On Monday, 24 February 2014 at 22:14:34 UTC, Dejan Lekic wrote: No criticism should stop this module being accepted, as we do not have any other lexer in the runtime anyway. Therefore I suggest we accept std.lexer until a better solution comes

Re: Formal review of std.lexer

2014-02-24 Thread Meta
On Monday, 24 February 2014 at 22:14:34 UTC, Dejan Lekic wrote: No criticism should stop this module being accepted, as we do not have any other lexer in the runtime anyway. Therefore I suggest we accept std.lexer until a better solution comes up. Naturally anyone should be encouraged to provid

Re: Formal review of std.lexer

2014-02-24 Thread Dejan Lekic
On Friday, 21 February 2014 at 12:12:17 UTC, Dicebot wrote: http://wiki.dlang.org/Review/std.lexer This is follow-up by Brian to his earlier proposal (http://wiki.dlang.org/Review/std.d.lexer). This time proposed module focuses instead on generic lexer generation as discussed in matching voti

Re: Formal review of std.lexer

2014-02-24 Thread Dicebot
Brian, do you have benchmarks for this proposal similar to ones you have provided in old review threads? (vs DMD frontend lexer)

Re: Formal review of std.lexer

2014-02-22 Thread Marc Schütz
On Friday, 21 February 2014 at 22:13:30 UTC, Andrej Mitrovic wrote: On 2/21/14, Joseph Cassman wrote: 1. StringCache is a custom hash table. It looks like it's primary role is to reduce some sort of duplication. It's used for string interning. But that still doesn't explain why a custom ha

Re: Formal review of std.lexer

2014-02-21 Thread Brian Schott
On Friday, 21 February 2014 at 22:14:36 UTC, Andrej Mitrovic wrote: Hmm, maybe I should make a pull request upstream? You should.

Re: Formal review of std.lexer

2014-02-21 Thread Andrei Alexandrescu
On 2/21/14, 9:51 PM, Brian Schott wrote: Does this mean that you're finally getting approval to release your lexer generator? Affirmative. It'll happen Real Soon Now(tm). Andrei

Re: Formal review of std.lexer

2014-02-21 Thread Andrej Mitrovic
On 2/21/14, Andrej Mitrovic wrote: > I've used it to learn how to use DLexer, here's my more-documented > version of the above if it helps anyone, which also takes the D > parser's whitespace function and uses that in order to track column > numbers, it injects a toString like I've mentioned, and

Re: Formal review of std.lexer

2014-02-21 Thread Andrej Mitrovic
On 2/21/14, Joseph Cassman wrote: > 1. StringCache is a custom hash table. It looks like it's primary > role is to reduce some sort of duplication. It's used for string interning. > 3. I tried to understand the reason for and usage of the > "extraFields" parameter in "TokenStructure" but couldn'

Re: Formal review of std.lexer

2014-02-21 Thread Brian Schott
Does this mean that you're finally getting approval to release your lexer generator? On Friday, 21 February 2014 at 17:06:23 UTC, Andrei Alexandrescu wrote: Can we please defer this by one week? Thanks, Andrei

Re: Formal review of std.lexer

2014-02-21 Thread Dicebot
On Friday, 21 February 2014 at 17:15:41 UTC, Jacob Carlborg wrote: On 2014-02-21 18:06, Andrei Alexandrescu wrote: Can we please defer this by one week? Just make the review period one week longer. This. There is no real rationale behind existing default review period so extending it until

Re: Formal review of std.lexer

2014-02-21 Thread Jacob Carlborg
On 2014-02-21 18:06, Andrei Alexandrescu wrote: Can we please defer this by one week? Just make the review period one week longer. -- /Jacob Carlborg

Re: Formal review of std.lexer

2014-02-21 Thread Andrei Alexandrescu
On 2/21/14, 2:12 PM, Dicebot wrote: http://wiki.dlang.org/Review/std.lexer This is follow-up by Brian to his earlier proposal (http://wiki.dlang.org/Review/std.d.lexer). This time proposed module focuses instead on generic lexer generation as discussed in matching voting thread. Docs: http://ha

Re: Formal review of std.lexer

2014-02-21 Thread Joseph Cassman
On Friday, 21 February 2014 at 12:12:17 UTC, Dicebot wrote: http://wiki.dlang.org/Review/std.lexer This is follow-up by Brian to his earlier proposal (http://wiki.dlang.org/Review/std.d.lexer). This time proposed module focuses instead on generic lexer generation as discussed in matching voti

Re: Formal review of std.lexer

2014-02-21 Thread Vladimir Panteleev
On Friday, 21 February 2014 at 12:12:17 UTC, Dicebot wrote: http://wiki.dlang.org/Review/std.lexer First of all, thank you for the great work. This is a very important project. I'll begin with reviewing the documentation. Summary Some simple explanation of the terminology and concepts wo

Formal review of std.lexer

2014-02-21 Thread Dicebot
http://wiki.dlang.org/Review/std.lexer This is follow-up by Brian to his earlier proposal (http://wiki.dlang.org/Review/std.d.lexer). This time proposed module focuses instead on generic lexer generation as discussed in matching voting thread. Docs: http://hackerpilot.github.io/experimental