Re: [sqlite] Lemon not reducing
For those reading this thread, I could solve my problme by using ragel (http://www.cs.queensu.ca/~thurston/ragel/). You can define leaving actions but also 'any change' actions. It was also easier to include in a C++ project then lemon/flex. If my grammar becomes more complicated, I heard it is possible to make a good collaboration between ragel and lemon (ragel is then used as a tokenizer). 2007/10/25, Gaspard Bucher <[EMAIL PROTECTED]>: > > I do not understand why lemon waits for one more token when it has > > enough information to reduce > ... > > > > >>> I don't think you can. Why do you want to? Why not just go > > >>> ahead and send it the next token? > > >>> > > >> Most people find a way around this problem using white-space. This > > >> could be a solution but then my grammar will be filled with > > >> "white-space | nothing" rules and I thought Lemon could reduce when > > >> there is no other way out of the current stack as it is more elegant. > > >> > > >> > > LA(LR) is the answer - just drop Your's tokens as their arrive and give > > a chance for the parser to be LALR, not LR or SLR :) > > > > mak > > > Lookahead improves what we can do with a grammar, but when there is no > ambiguity, it should resolve without waiting for the next token. If > your grammar is so simple it's an SLR, why not treat it so and ease > the creation of grammars without needing to add "separator" tokens > just so there is a lookahead (most languages use ";" or white space > for that purpose). > > My grammar works with all "whitespace or nothing" rules, but I have to > be careful to avoid conflicts. When I write it without the whitespace > stuff (eaten in the tokenizer), I have no conflict. Simple example: > main ::= commands. > commands ::= . > commands ::= commands ws command. > command ::= variable EQUAL value. > variable ::= ws IDENTIFIER ws. > value::= ws NUMBER ws. > ws ::= . > ws ::= WHITE. > > 2 parsing conflicts. > > Whithout whitespace stuff: > main ::= commands. > commands ::= . > commands ::= commands command. > command ::= variable EQUAL value. > variable ::= IDENTIFIER. > value::= NUMBER. > > no conflicts, easier and cleaner. > > I know how to fix the first grammar, but you have to think of the > actual succession to avoid two "ws" from coming next to the other, > this thinking is not really related to the grammar from my point of > view. > > Gaspard > - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Lemon not reducing
> I do not understand why lemon waits for one more token when it has > enough information to reduce ... > > >>> I don't think you can. Why do you want to? Why not just go > >>> ahead and send it the next token? > >>> > >> Most people find a way around this problem using white-space. This > >> could be a solution but then my grammar will be filled with > >> "white-space | nothing" rules and I thought Lemon could reduce when > >> there is no other way out of the current stack as it is more elegant. > >> > >> > LA(LR) is the answer - just drop Your's tokens as their arrive and give > a chance for the parser to be LALR, not LR or SLR :) > > mak > Lookahead improves what we can do with a grammar, but when there is no ambiguity, it should resolve without waiting for the next token. If your grammar is so simple it's an SLR, why not treat it so and ease the creation of grammars without needing to add "separator" tokens just so there is a lookahead (most languages use ";" or white space for that purpose). My grammar works with all "whitespace or nothing" rules, but I have to be careful to avoid conflicts. When I write it without the whitespace stuff (eaten in the tokenizer), I have no conflict. Simple example: main ::= commands. commands ::= . commands ::= commands ws command. command ::= variable EQUAL value. variable ::= ws IDENTIFIER ws. value::= ws NUMBER ws. ws ::= . ws ::= WHITE. > 2 parsing conflicts. Whithout whitespace stuff: main ::= commands. commands ::= . commands ::= commands command. command ::= variable EQUAL value. variable ::= IDENTIFIER. value::= NUMBER. > no conflicts, easier and cleaner. I know how to fix the first grammar, but you have to think of the actual succession to avoid two "ws" from coming next to the other, this thinking is not really related to the grammar from my point of view. Gaspard - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Lemon not reducing
Gaspard Bucher wrote: > PS: There is another reason, aside from aesthetics and simpler grammar > to filter white spaces inside the tokenizer: you avoid all the parser > conflicts you could get with "empty | or space" rules. > > 2007/10/24, Gaspard Bucher <[EMAIL PROTECTED]>: > >>> Gaspard Bucher <[EMAIL PROTECTED]> wrote: >>> I do not understand why lemon waits for one more token when it has enough information to reduce. I want to recognize : foo = Bar() when the token CLOSE_PAR is received, not when an extra token is parsed.. How can I avoid lemon waiting for the extra token before reducing ? >>> I don't think you can. Why do you want to? Why not just go >>> ahead and send it the next token? >>> >> Most people find a way around this problem using white-space. This >> could be a solution but then my grammar will be filled with >> "white-space | nothing" rules and I thought Lemon could reduce when >> there is no other way out of the current stack as it is more elegant. >> I went into the sources and saw this comment: >> >> LA(LR) is the answer - just drop Your's tokens as their arrive and give a chance for the parser to be LALR, not LR or SLR :) mak - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Lemon not reducing
PS: There is another reason, aside from aesthetics and simpler grammar
to filter white spaces inside the tokenizer: you avoid all the parser
conflicts you could get with "empty | or space" rules.
2007/10/24, Gaspard Bucher <[EMAIL PROTECTED]>:
> > Gaspard Bucher <[EMAIL PROTECTED]> wrote:
> > > I do not understand why lemon waits for one more token when it has
> > > enough information to reduce.
> > >
> > > I want to recognize :
> > > foo = Bar()
> > > when the token CLOSE_PAR is received, not when an extra token is parsed..
> > >
> > > How can I avoid lemon waiting for the extra token before reducing ?
> > >
> >
> > I don't think you can. Why do you want to? Why not just go
> > ahead and send it the next token?
> Most people find a way around this problem using white-space. This
> could be a solution but then my grammar will be filled with
> "white-space | nothing" rules and I thought Lemon could reduce when
> there is no other way out of the current stack as it is more elegant.
> I went into the sources and saw this comment:
>
> I think I might find something to do here:
> in yy_find_shift_action :
> when we find the action:
>
> // return yy_action[i];
> result = yy_action[i];
> printf("==> next : %i\n", result);
> printf("==> next_next: %i\n", yy_shift_ofst[result % (YYNSTATE+YYNRULE)]);
> // if the shift results in a default action do this action now.
> if (result < (YYNSTATE+YYNRULE) && yy_shift_ofst[result] ==
> YY_SHIFT_USE_DFLT) {
> // do the change ourself
>
> YYMINORTYPE yyminorunion;
> yy_shift(pParser,result,iLookAhead,);
> pParser->yyerrcnt--;
> return yy_default[result];
> } else
> return result;
>
> This works except when the reduction reaches state '1'. Somehow, it
> should jump directly to state '0', clearing the current lookahead.
>
> > --
> > D. Richard Hipp <[EMAIL PROTECTED]>
> >
> >
> > -
> > To unsubscribe, send email to [EMAIL PROTECTED]
> > -
> >
> >
> > !DSPAM:471f82d0320381804284693!
> >
> >
>
-
To unsubscribe, send email to [EMAIL PROTECTED]
-
Re: [sqlite] Lemon not reducing
> Gaspard Bucher <[EMAIL PROTECTED]> wrote:
> > I do not understand why lemon waits for one more token when it has
> > enough information to reduce.
> >
> > I want to recognize :
> > foo = Bar()
> > when the token CLOSE_PAR is received, not when an extra token is parsed..
> >
> > How can I avoid lemon waiting for the extra token before reducing ?
> >
>
> I don't think you can. Why do you want to? Why not just go
> ahead and send it the next token?
Most people find a way around this problem using white-space. This
could be a solution but then my grammar will be filled with
"white-space | nothing" rules and I thought Lemon could reduce when
there is no other way out of the current stack as it is more elegant.
I went into the sources and saw this comment:
I think I might find something to do here:
in yy_find_shift_action :
when we find the action:
// return yy_action[i];
result = yy_action[i];
printf("==> next : %i\n", result);
printf("==> next_next: %i\n", yy_shift_ofst[result % (YYNSTATE+YYNRULE)]);
// if the shift results in a default action do this action now.
if (result < (YYNSTATE+YYNRULE) && yy_shift_ofst[result] ==
YY_SHIFT_USE_DFLT) {
// do the change ourself
YYMINORTYPE yyminorunion;
yy_shift(pParser,result,iLookAhead,);
pParser->yyerrcnt--;
return yy_default[result];
} else
return result;
This works except when the reduction reaches state '1'. Somehow, it
should jump directly to state '0', clearing the current lookahead.
> --
> D. Richard Hipp <[EMAIL PROTECTED]>
>
>
> -
> To unsubscribe, send email to [EMAIL PROTECTED]
> -
>
>
> !DSPAM:471f82d0320381804284693!
>
>
-
To unsubscribe, send email to [EMAIL PROTECTED]
-
Re: [sqlite] Lemon not reducing
Gaspard Bucher <[EMAIL PROTECTED]> wrote: > I do not understand why lemon waits for one more token when it has > enough information to reduce. > > I want to recognize : > foo = Bar() > when the token CLOSE_PAR is received, not when an extra token is parsed. > > How can I avoid lemon waiting for the extra token before reducing ? > I don't think you can. Why do you want to? Why not just go ahead and send it the next token? -- D. Richard Hipp <[EMAIL PROTECTED]> - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] Lemon not reducing
I do not understand why lemon waits for one more token when it has
enough information to reduce.
I want to recognize :
foo = Bar()
when the token CLOSE_PAR is received, not when an extra token is parsed.
How can I avoid lemon waiting for the extra token before reducing ?
Thanks for your help !
Gaspard
=== GRAMMAR ==
main::= commands.
commands::= . /* can be empty */
commands::= WHITE_SPACE.
commands::= commands command. /* many commands */
command ::= variable EQUAL class parameters. { printf("[new
object]\n"); }
variable::= IDENTIFIER.
class ::= CONST_IDENTIFIER.
parameters ::= OPEN_PAR CLOSE_PAR.
=== DEBUG OUTPUT ==
Parser started.
> foo = Bar()
>>Input IDENTIFIER
>>Reduce [commands ::=].
>>Shift 1
>>Stack: commands
>>Shift 21
>>Stack: commands IDENTIFIER
>>Input EQUAL
>>Reduce [variable ::= IDENTIFIER].
>>Shift 3
>>Stack: commands variable
>>Shift 4
>>Stack: commands variable EQUAL
>>Input CONST_IDENTIFIER
>>Shift 20
>>Stack: commands variable EQUAL CONST_IDENTIFIER
>>Input OPEN_PAR
>>Reduce [class ::= CONST_IDENTIFIER].
>>Shift 5
>>Stack: commands variable EQUAL class
>>Shift 7
>>Stack: commands variable EQUAL class OPEN_PAR
>>Input CLOSE_PAR
>>Shift 8
>>Stack: commands variable EQUAL class OPEN_PAR CLOSE_PAR
At this point, lemon has all the information needed, but it waits
before reducing...
---
http://rubyk.org, scriptable multimedia controller
-
To unsubscribe, send email to [EMAIL PROTECTED]
-
