ID: 45372 Updated by: [EMAIL PROTECTED] Reported By: [EMAIL PROTECTED] -Status: Closed +Status: Open Bug Type: Scripting Engine problem Operating System: linux PHP Version: 5.3CVS-2008-06-27 (CVS) Assigned To: helly New Comment:
This is not fixed, actually (is it OK to change Status back to Open?). Nuno, I saw your commit yesterday, which didn't seem like it would help (as it wasn't related to what I said above), but wanted to wait until I could check again to make sure I wasn't crazy with my above description. :-) I just tried the latest Windows snapshot and it's still generating a parse error with the *single line file* (no newline at the end, which .+ won't match, therefore won't trigger the YYFILL() "return 0" thing) descibed in this report (and the CLI example in Bug #44654). Can't be only broken on Windows since everything uses the same generated scanner code... Something like Alan's scanning loop could be done after just matching #, BUT that's just another workaround for that underlying re2c/YYFILL() problem (also affecting other things). I believe if you use the tokenizer extension, you can see that if the last token of code is matched by a variable length rule, it won't be returned. e.g. my example of a simple rule, [a-z]+ not matching input "foo" Previous Comments: ------------------------------------------------------------------------ [2008-07-06 17:01:55] [EMAIL PROTECTED] This bug has been fixed in CVS. Snapshots of the sources are packaged every three hours; this change will be in the next snapshot. You can grab the snapshot at http://snaps.php.net/. Thank you for the report, and for helping us make PHP better. ------------------------------------------------------------------------ [2008-06-27 14:57:57] [EMAIL PROTECTED] Not sure why re2c needs to deal with the #bang situation looking at the code it would be better to eat that line outside of the lexer.. Something like: int ini_lex(zval *ini_lval TSRMLS_DC) { if ((YYCTYPE*)yytext == SCNG(yy_start) && *yych == '#') { while(*yych != '\n' && *yych != '\n' && yych < yyend) { yych++; } while((*yych == '\n' || *yych == '\n') && yych < yyend) { yych++; } YYCURSOR = yych; } ..... ------------------------------------------------------------------------ [2008-06-27 11:26:18] [EMAIL PROTECTED] Duplicated... Bug #45147 ------------------------------------------------------------------------ [2008-06-27 09:31:21] [EMAIL PROTECTED] This should work like in older releases, Marcus please check it! ------------------------------------------------------------------------ [2008-06-27 09:05:48] [EMAIL PROTECTED] (yyless(1) could just be used before the goto...) Anyway, did you actually try that? AFAIK it still won't work, at least with your single line example (which there's already been at least one report about). While the local code fix is correct, the re2c code/logic seems flawed to me. (Maybe this bug report can be about that instead, in general, since I didn't get around to sending a follow-up message to the internals@ list yet, explaining things. :-)) In this example, it will still be broken because of the YYFILL() check -- each time it checks if the next character can match, even when it's at the end of the input. YYFILL() then makes it return, completely ignoring anything that has matched up to that point! I'm not sure if this explanation is 100% correct, but I believe this wrong behavior happens when EOF is encountered while trying to match the variable length part of ANY rule; or something close to that. :-) It's been over a month since I tried to track and figure out what was happening. Granted, most of the cases (unlike yours), where the match is aborted because of YYFILL(), it's with invalid code, but it shouldn't happen. BTW, I think the part with the inline_char_handler label where it looks for opening PHP tags in the HTML, while a good optimization (using memchr() to find < etc.), was actually added as a workaround for this re2c/YYFILL() behavior. I didn't try it, but from what I've observed, I think whatever plain HTML was at the end of a file would have been lost if a regular rule (like in Flex) was used to match it... Oh, there are also some more bugs in the code that looks for opening PHP tag, but they wouldn't be found as easily as this (and haven't been reported so far). I think I know how it can be fixed nicely, along with some more other scanner optimizations (for inline HTML and comments, basically). But I haven't done anything yet since some of it won't even work with these re2c/YYFILL() issues. :-/ Finally, to simplify what I think is the basic, underlying flaw with the code of re2c and YYFILL() now, here's a super easy example. Say you have one rule: [a-z]+ It will NEVER match any input that a person would think, such as the string "foo" -- seems pretty messed up to me!? ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at http://bugs.php.net/45372 -- Edit this bug report at http://bugs.php.net/?id=45372&edit=1
