Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-18 Thread Hans Åberg
> On 18 Feb 2019, at 06:44, Akim Demaille wrote: > >> Le 18 févr. 2019 à 00:10, Hans Åberg a écrit : >> >>> On 17 Feb 2019, at 23:10, Peng Yu wrote: >>> >>> This lexical tie-in creates feedback from the parser to the lexer. So >>> the lexer cannot be tested standalone. >>> >>> But the princ

Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Akim Demaille
> Le 18 févr. 2019 à 00:10, Hans Åberg a écrit : > > >> On 17 Feb 2019, at 23:10, Peng Yu wrote: >> >> This lexical tie-in creates feedback from the parser to the lexer. So >> the lexer cannot be tested standalone. >> >> But the principle of separating lexer and parser is to make parser >>

Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Akim Demaille
Hi Peng! > Le 17 févr. 2019 à 23:10, Peng Yu a écrit : > > This lexical tie-in creates feedback from the parser to the lexer. So > the lexer cannot be tested standalone. Well, yes, it can, but that's not as elegant, agreed. > But the principle of separating lexer and parser is to make parser >

Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Hans Åberg
> On 17 Feb 2019, at 23:10, Peng Yu wrote: > > This lexical tie-in creates feedback from the parser to the lexer. So > the lexer cannot be tested standalone. > > But the principle of separating lexer and parser is to make parser > builtin on top of the parser. Is there something that can avoid

Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Peng Yu
This lexical tie-in creates feedback from the parser to the lexer. So the lexer cannot be tested standalone. But the principle of separating lexer and parser is to make parser builtin on top of the parser. Is there something that can avoid the feedback yet still allow context-dependent parsing? Al

Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Simon Richter
Hi, On 17.02.19 14:08, Peng Yu wrote: > [[:alpha:]_][[:alnum:]_]=[[:digit:]+] { /* parse yytext to get the > name and value, then do the assignment */ } The main reason you are using a lexer is to avoid writing code for manual parsing. The lexer can already tell you where the equals sign is and

Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Akim Demaille
Hi Peng, > Le 17 févr. 2019 à 17:36, Peng Yu a écrit : > > Sometimes, the best implementation may not be what it obviously should > look like. I think that there can be cases in which more actions > should be in the lexer instead of the parser. Yes, I agree. > Consider the parameter expansion

Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Hans Åberg
> On 17 Feb 2019, at 17:36, Peng Yu wrote: > > But how to recognize the nested parameter expansion assignment in the > first place? The lexer should have builtin states to capture paired > `{` `}`, and use states to remember whether it is in substring > extraction or pattern replacement in orde

Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Peng Yu
Sometimes, the best implementation may not be what it obviously should look like. I think that there can be cases in which more actions should be in the lexer instead of the parser. Consider the parameter expansion (along with assignment) in bash. https://www.gnu.org/software/bash/manual/html_nod

Re: How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Akim Demaille
Hi! > Le 17 févr. 2019 à 14:08, Peng Yu a écrit : > > Hi, > > The more I study flex/bison, the more confused I got about when to use > what actions and what to put in lexer and what to put in grammar. Usually it's quite clear: build words from letters in the scanner, build sentences from words

How to decide what to put in the lexer and the grammar respectively?

2019-02-17 Thread Peng Yu
Hi, The more I study flex/bison, the more confused I got about when to use what actions and what to put in lexer and what to put in grammar. For example, for an assignment, x=10 it can be processed by the lexer, [[:alpha:]_][[:alnum:]_]=[[:digit:]+] { /* parse yytext to get the name and value