[webkit-dev] Should we close the tree? (was: Re: the new TestExpectations syntax is landing soon)
Now tracked as https://bugs.webkit.org/show_bug.cgi?id=97182. I think that we should close the tree if resolving this takes any significant time. Not being able to see how exactly tests are failing on other platforms is unacceptable. - WBR, Alexey Proskuryakov 20.09.2012, в 3:54, Osztrogonac Csaba o...@inf.u-szeged.hu написал(а): Unfortunately r129047 broke the results.html, see https://bugs.webkit.org/show_bug.cgi?id=96845#c9 for details. Dirk Pranke írta: These changes are now starting to land ... as of r129047, TEXT, IMAGE+TEXT, and AUDIO are no longer legal keywords in the TestExpectations syntax ... you should use FAIL instead. I will be landing the support for the new syntax as quickly as I can to minimize the transition period. Apologies for the inconvenience. -- Dirk On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, The new format of the much-debated TestExpectations syntax will be landing soon (hopefully in the next couple days). For those of who have forgotten / repressed the earlier debates, the new syntax looks something like: webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [ ImageOnlyFailure ] Andis documented in full at https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded . ( The [ and ] characters are delimiters, not EBNF optional markers, although those sections are in fact optional :) ). Note that the new syntax means that Skipped files are a syntactic subset of TestExpectations files, and I plan to convert all of the Skipped files to TestExpectations files via copy and paste shortly after the new syntax is landed, and then drop support for Skipped files (I will update ORWT to use the new files and treat any entry as a Skip). The plan for landing these changes is: 1) Add support for parsing the new lines and converting them back into the old format (internally) so that both syntaxes are supported 2) Convert all the existing files over 3) Make sure things aren't broken :) 4) Drop support for the old syntax I plan for this to all happen quickly, in less than a day. This means that if you have patches posted that modify those files they may become stale and need to be updated. Changes from the old syntax: 1. We use URLs (a specific whitelisted set; let me know if you want to add to it) instead of BUGWK12345 etc. 2. We use bug(dpranke) instead of BUGDPRANKE 3. We use CamelCase instead of SHOUTING 4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and AUDIO - these failures will be indistinguishable in the new world, meaning that you can't distinguish between text only and both image and text. Since only Chromium runs pixel tests by default, this shouldn't be a big deal. 5. We use ImageOnlyFailure to represent what used to be IMAGE 6. We use [ and ] for delimiters instead of : and = 7. We use # instead of // as a comment 7. WontFix will now imply Skip, i.e., tests marked WontFix will automatically be Skipped 8. WontFix and Skip will not require (or even allow) any other expectations, i.e., you can't say [ WontFix Crash ]. If you want to indicate that the test will crash if you actually run it, use a comment. 9. WontFix, Skip, Slow, and Rebaseline all move from the left hand side to the right. The only keywords on the left restrict which configurations the lines apply to. I will send out follow-up emails as this stuff lands. Please let me know if you have any questions. Thanks! -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Should we close the tree? (was: Re: the new TestExpectations syntax is landing soon)
I'd prefer to see the patch rolled out. Geoff On Sep 20, 2012, at 11:07 AM, Alexey Proskuryakov a...@webkit.org wrote: Now tracked as https://bugs.webkit.org/show_bug.cgi?id=97182. I think that we should close the tree if resolving this takes any significant time. Not being able to see how exactly tests are failing on other platforms is unacceptable. - WBR, Alexey Proskuryakov 20.09.2012, в 3:54, Osztrogonac Csaba o...@inf.u-szeged.hu написал(а): Unfortunately r129047 broke the results.html, see https://bugs.webkit.org/show_bug.cgi?id=96845#c9 for details. Dirk Pranke írta: These changes are now starting to land ... as of r129047, TEXT, IMAGE+TEXT, and AUDIO are no longer legal keywords in the TestExpectations syntax ... you should use FAIL instead. I will be landing the support for the new syntax as quickly as I can to minimize the transition period. Apologies for the inconvenience. -- Dirk On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, The new format of the much-debated TestExpectations syntax will be landing soon (hopefully in the next couple days). For those of who have forgotten / repressed the earlier debates, the new syntax looks something like: webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [ ImageOnlyFailure ] Andis documented in full at https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded . ( The [ and ] characters are delimiters, not EBNF optional markers, although those sections are in fact optional :) ). Note that the new syntax means that Skipped files are a syntactic subset of TestExpectations files, and I plan to convert all of the Skipped files to TestExpectations files via copy and paste shortly after the new syntax is landed, and then drop support for Skipped files (I will update ORWT to use the new files and treat any entry as a Skip). The plan for landing these changes is: 1) Add support for parsing the new lines and converting them back into the old format (internally) so that both syntaxes are supported 2) Convert all the existing files over 3) Make sure things aren't broken :) 4) Drop support for the old syntax I plan for this to all happen quickly, in less than a day. This means that if you have patches posted that modify those files they may become stale and need to be updated. Changes from the old syntax: 1. We use URLs (a specific whitelisted set; let me know if you want to add to it) instead of BUGWK12345 etc. 2. We use bug(dpranke) instead of BUGDPRANKE 3. We use CamelCase instead of SHOUTING 4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and AUDIO - these failures will be indistinguishable in the new world, meaning that you can't distinguish between text only and both image and text. Since only Chromium runs pixel tests by default, this shouldn't be a big deal. 5. We use ImageOnlyFailure to represent what used to be IMAGE 6. We use [ and ] for delimiters instead of : and = 7. We use # instead of // as a comment 7. WontFix will now imply Skip, i.e., tests marked WontFix will automatically be Skipped 8. WontFix and Skip will not require (or even allow) any other expectations, i.e., you can't say [ WontFix Crash ]. If you want to indicate that the test will crash if you actually run it, use a comment. 9. WontFix, Skip, Slow, and Rebaseline all move from the left hand side to the right. The only keywords on the left restrict which configurations the lines apply to. I will send out follow-up emails as this stuff lands. Please let me know if you have any questions. Thanks! -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Should we close the tree? (was: Re: the new TestExpectations syntax is landing soon)
Fixing the problem will likely take less time at this point than rolling the patch out as there are a number of dependent patches. The fix should be in soon. On Thu, Sep 20, 2012 at 11:10 AM, Geoffrey Garen gga...@apple.com wrote: I'd prefer to see the patch rolled out. Geoff On Sep 20, 2012, at 11:07 AM, Alexey Proskuryakov a...@webkit.org wrote: Now tracked as https://bugs.webkit.org/show_bug.cgi?id=97182. I think that we should close the tree if resolving this takes any significant time. Not being able to see how exactly tests are failing on other platforms is unacceptable. - WBR, Alexey Proskuryakov 20.09.2012, в 3:54, Osztrogonac Csaba o...@inf.u-szeged.hu написал(а): Unfortunately r129047 broke the results.html, see https://bugs.webkit.org/show_bug.cgi?id=96845#c9 for details. Dirk Pranke írta: These changes are now starting to land ... as of r129047, TEXT, IMAGE+TEXT, and AUDIO are no longer legal keywords in the TestExpectations syntax ... you should use FAIL instead. I will be landing the support for the new syntax as quickly as I can to minimize the transition period. Apologies for the inconvenience. -- Dirk On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, The new format of the much-debated TestExpectations syntax will be landing soon (hopefully in the next couple days). For those of who have forgotten / repressed the earlier debates, the new syntax looks something like: webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [ ImageOnlyFailure ] Andis documented in full at https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded . ( The [ and ] characters are delimiters, not EBNF optional markers, although those sections are in fact optional :) ). Note that the new syntax means that Skipped files are a syntactic subset of TestExpectations files, and I plan to convert all of the Skipped files to TestExpectations files via copy and paste shortly after the new syntax is landed, and then drop support for Skipped files (I will update ORWT to use the new files and treat any entry as a Skip). The plan for landing these changes is: 1) Add support for parsing the new lines and converting them back into the old format (internally) so that both syntaxes are supported 2) Convert all the existing files over 3) Make sure things aren't broken :) 4) Drop support for the old syntax I plan for this to all happen quickly, in less than a day. This means that if you have patches posted that modify those files they may become stale and need to be updated. Changes from the old syntax: 1. We use URLs (a specific whitelisted set; let me know if you want to add to it) instead of BUGWK12345 etc. 2. We use bug(dpranke) instead of BUGDPRANKE 3. We use CamelCase instead of SHOUTING 4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and AUDIO - these failures will be indistinguishable in the new world, meaning that you can't distinguish between text only and both image and text. Since only Chromium runs pixel tests by default, this shouldn't be a big deal. 5. We use ImageOnlyFailure to represent what used to be IMAGE 6. We use [ and ] for delimiters instead of : and = 7. We use # instead of // as a comment 7. WontFix will now imply Skip, i.e., tests marked WontFix will automatically be Skipped 8. WontFix and Skip will not require (or even allow) any other expectations, i.e., you can't say [ WontFix Crash ]. If you want to indicate that the test will crash if you actually run it, use a comment. 9. WontFix, Skip, Slow, and Rebaseline all move from the left hand side to the right. The only keywords on the left restrict which configurations the lines apply to. I will send out follow-up emails as this stuff lands. Please let me know if you have any questions. Thanks! -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Should we close the tree? (was: Re: the new TestExpectations syntax is landing soon)
This should be fixed now. I apologize for the problems. As Ojan said, rolling it out would've been tricky because the changes to the expectation syntax depended on this change. In retrospect it might've been better to have more time elapse between the initial change and the changes to the syntax but I didn't want to impose multiple changes to the format on people for an extended period of time. Some might wonder why this got through ... we had unit tests for both the python side and the html side of this, but not an integration test that ensured that the right thing went all the way through (it's not actually clear how to test this without being overly invasive on your checkout, but I'm going to keep thinking about it). Unfortunately, this also wasn't caught in a review and the tree happened to be fairly green yesterday so I missed it in garden-o-matic and on the bots (which don't really use results.html anyway) :(. -- Dirk On Thu, Sep 20, 2012 at 11:14 AM, Ojan Vafai o...@chromium.org wrote: Fixing the problem will likely take less time at this point than rolling the patch out as there are a number of dependent patches. The fix should be in soon. On Thu, Sep 20, 2012 at 11:10 AM, Geoffrey Garen gga...@apple.com wrote: I'd prefer to see the patch rolled out. Geoff On Sep 20, 2012, at 11:07 AM, Alexey Proskuryakov a...@webkit.org wrote: Now tracked as https://bugs.webkit.org/show_bug.cgi?id=97182. I think that we should close the tree if resolving this takes any significant time. Not being able to see how exactly tests are failing on other platforms is unacceptable. - WBR, Alexey Proskuryakov 20.09.2012, в 3:54, Osztrogonac Csaba o...@inf.u-szeged.hu написал(а): Unfortunately r129047 broke the results.html, see https://bugs.webkit.org/show_bug.cgi?id=96845#c9 for details. Dirk Pranke írta: These changes are now starting to land ... as of r129047, TEXT, IMAGE+TEXT, and AUDIO are no longer legal keywords in the TestExpectations syntax ... you should use FAIL instead. I will be landing the support for the new syntax as quickly as I can to minimize the transition period. Apologies for the inconvenience. -- Dirk On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, The new format of the much-debated TestExpectations syntax will be landing soon (hopefully in the next couple days). For those of who have forgotten / repressed the earlier debates, the new syntax looks something like: webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [ ImageOnlyFailure ] Andis documented in full at https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded . ( The [ and ] characters are delimiters, not EBNF optional markers, although those sections are in fact optional :) ). Note that the new syntax means that Skipped files are a syntactic subset of TestExpectations files, and I plan to convert all of the Skipped files to TestExpectations files via copy and paste shortly after the new syntax is landed, and then drop support for Skipped files (I will update ORWT to use the new files and treat any entry as a Skip). The plan for landing these changes is: 1) Add support for parsing the new lines and converting them back into the old format (internally) so that both syntaxes are supported 2) Convert all the existing files over 3) Make sure things aren't broken :) 4) Drop support for the old syntax I plan for this to all happen quickly, in less than a day. This means that if you have patches posted that modify those files they may become stale and need to be updated. Changes from the old syntax: 1. We use URLs (a specific whitelisted set; let me know if you want to add to it) instead of BUGWK12345 etc. 2. We use bug(dpranke) instead of BUGDPRANKE 3. We use CamelCase instead of SHOUTING 4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and AUDIO - these failures will be indistinguishable in the new world, meaning that you can't distinguish between text only and both image and text. Since only Chromium runs pixel tests by default, this shouldn't be a big deal. 5. We use ImageOnlyFailure to represent what used to be IMAGE 6. We use [ and ] for delimiters instead of : and = 7. We use # instead of // as a comment 7. WontFix will now imply Skip, i.e., tests marked WontFix will automatically be Skipped 8. WontFix and Skip will not require (or even allow) any other expectations, i.e., you can't say [ WontFix Crash ]. If you want to indicate that the test will crash if you actually run it, use a comment. 9. WontFix, Skip, Slow, and Rebaseline all move from the left hand side to the right. The only keywords on the left restrict which configurations the lines apply to. I will send out follow-up emails as this stuff lands. Please let me know if you have any questions. Thanks! -- Dirk
