Re: Writing a JFlex lexer for D - have an issue with cycles

2017-01-23 Thread Profile Anaysis via Digitalmars-d
On Monday, 23 January 2017 at 01:46:58 UTC, FatalCatharsis wrote: On Monday, 23 January 2017 at 00:46:30 UTC, Profile Anaysis wrote: The real issue is ambiguity. Any time you have a cycle you must be able to get out of it and so your rules must be organized so that one always checks to see if

Re: Writing a JFlex lexer for D - have an issue with cycles

2017-01-22 Thread Basile B. via Digitalmars-d
On Monday, 23 January 2017 at 00:03:44 UTC, FatalCatharsis wrote: On Sunday, 22 January 2017 at 23:20:27 UTC, Basile B. wrote: One way to do this is not to do anything special for q{. [...] This is for tooling, but I still want it to be accurate. Are you suggesting I could move handling

Re: Writing a JFlex lexer for D - have an issue with cycles

2017-01-22 Thread FatalCatharsis via Digitalmars-d
On Monday, 23 January 2017 at 01:40:28 UTC, Chris Wright wrote: On Mon, 23 Jan 2017 00:46:30 +, Profile Anaysis wrote: But this is ok because parsers are recursive in nature, so you just have to have your rule be able to terminate in a logical way. Except it's part of the lexer, and most

Re: Writing a JFlex lexer for D - have an issue with cycles

2017-01-22 Thread FatalCatharsis via Digitalmars-d
On Monday, 23 January 2017 at 00:46:30 UTC, Profile Anaysis wrote: The real issue is ambiguity. Any time you have a cycle you must be able to get out of it and so your rules must be organized so that one always checks to see if termination has occurred before checking for nesting. If you

Re: Writing a JFlex lexer for D - have an issue with cycles

2017-01-22 Thread Chris Wright via Digitalmars-d
On Mon, 23 Jan 2017 00:46:30 +, Profile Anaysis wrote: > But this is ok because parsers are recursive in nature, so you just have > to have your rule be able to terminate in a logical way. Except it's part of the lexer, and most people (such as those creating the JFlex lexer generator)

Re: Writing a JFlex lexer for D - have an issue with cycles

2017-01-22 Thread Profile Anaysis via Digitalmars-d
On Sunday, 22 January 2017 at 22:11:08 UTC, FatalCatharsis wrote: I'm writing a flex lexer for D and I've hit a roadblock. It is almost working EXCEPT for one specific production. StringLiteral is cyclic and I don't know how to approach it. It is cyclic because: Token -> StringLiteral

Re: Writing a JFlex lexer for D - have an issue with cycles

2017-01-22 Thread FatalCatharsis via Digitalmars-d
On Sunday, 22 January 2017 at 23:20:27 UTC, Basile B. wrote: One way to do this is not to do anything special for q{. Just add a token for q{ and continue normal lexing. The token string content must be valid tokens so it should work. In facts it depends on what the scanner just be used

Re: Writing a JFlex lexer for D - have an issue with cycles

2017-01-22 Thread Basile B. via Digitalmars-d
On Sunday, 22 January 2017 at 22:11:08 UTC, FatalCatharsis wrote: I'm writing a flex lexer for D and I've hit a roadblock. It is almost working EXCEPT for one specific production. StringLiteral is cyclic and I don't know how to approach it. It is cyclic because: Token -> StringLiteral

Writing a JFlex lexer for D - have an issue with cycles

2017-01-22 Thread FatalCatharsis via Digitalmars-d
I'm writing a flex lexer for D and I've hit a roadblock. It is almost working EXCEPT for one specific production. StringLiteral is cyclic and I don't know how to approach it. It is cyclic because: Token -> StringLiteral -> TokenString -> Token To break the cycle, I was thinking I could