[webkit-dev] generic test expectations?
Just checking, but I don't see a way to add test expectations that apply generically (to all ports). It would be nice to have something like LayoutTests/platform/generic/TestExpectations to which one could add new tests that are known to fail everywhere (e.g., because the code that implements a feature that is tested by those tests is not yet committed), but which will (at some point in the near future) not fail (when the code that is to be tested is committed). At present, it seems that if one wishes to do this, then it is necessary to add entries to the each base port expectations (i.e., chromium, mac, win, etc), which is rather annoying. If there is no objection to adding such a generic platform expectations file, then I will undertake to do so. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
Just put the expected files in the same directory as the test. Simon On Nov 13, 2012, at 10:49 AM, Glenn Adams gl...@skynav.com wrote: Just checking, but I don't see a way to add test expectations that apply generically (to all ports). It would be nice to have something like LayoutTests/platform/generic/TestExpectations to which one could add new tests that are known to fail everywhere (e.g., because the code that implements a feature that is tested by those tests is not yet committed), but which will (at some point in the near future) not fail (when the code that is to be tested is committed). At present, it seems that if one wishes to do this, then it is necessary to add entries to the each base port expectations (i.e., chromium, mac, win, etc), which is rather annoying. If there is no objection to adding such a generic platform expectations file, then I will undertake to do so. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
That would seem to work only if the test(s) fail the same way on all ports. In the case that I'm working from, I'm using reftests, where I know the correct expectations, but the actual behavior will (does) differ on different ports (when the corresponding feature is committed). I would like to be able to (independently) commit new reftests *and* their known good expectation counterparts (that should apply on all ports), and then subsequently commit an independent patch that implements the expected behavior (on some but not all ports), and the comment further follow-on patches that fix the failing ports (possibly by writing new expectations for those specific ports). On Tue, Nov 13, 2012 at 11:52 AM, Ryosuke Niwa rn...@webkit.org wrote: It is customary to add a failing test expectation (i.e. *-expected.txt file that contains the said failure) in such cases. On Tue, Nov 13, 2012 at 10:49 AM, Glenn Adams gl...@skynav.com wrote: Just checking, but I don't see a way to add test expectations that apply generically (to all ports). It would be nice to have something like LayoutTests/platform/generic/TestExpectations to which one could add new tests that are known to fail everywhere (e.g., because the code that implements a feature that is tested by those tests is not yet committed), but which will (at some point in the near future) not fail (when the code that is to be tested is committed). At present, it seems that if one wishes to do this, then it is necessary to add entries to the each base port expectations (i.e., chromium, mac, win, etc), which is rather annoying. If there is no objection to adding such a generic platform expectations file, then I will undertake to do so. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 12:02 PM, Glenn Adams gl...@skynav.com wrote: implements the expected behavior (on some but not all ports), and the comment further follow-on s/the comment further/then commit further/ ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Perhaps we should discuss why you need them (in a separate thread with a separate subject line)? It sounds like this largely has to do with what features are enabled and/or supported? Perhaps said tests should just be skipped in the meantime? At any rate, we encourage people these days to check in expected failures rather than suppressing them using the TestExpectations files. I am hoping this week to finally get back to working on the -failing feature so you can at least distinguish expectations that are known to be failing/incorrect more easily. -- Dirk On Tue, Nov 13, 2012 at 11:02 AM, Glenn Adams gl...@skynav.com wrote: That would seem to work only if the test(s) fail the same way on all ports. In the case that I'm working from, I'm using reftests, where I know the correct expectations, but the actual behavior will (does) differ on different ports (when the corresponding feature is committed). I would like to be able to (independently) commit new reftests *and* their known good expectation counterparts (that should apply on all ports), and then subsequently commit an independent patch that implements the expected behavior (on some but not all ports), and the comment further follow-on patches that fix the failing ports (possibly by writing new expectations for those specific ports). On Tue, Nov 13, 2012 at 11:52 AM, Ryosuke Niwa rn...@webkit.org wrote: It is customary to add a failing test expectation (i.e. *-expected.txt file that contains the said failure) in such cases. On Tue, Nov 13, 2012 at 10:49 AM, Glenn Adams gl...@skynav.com wrote: Just checking, but I don't see a way to add test expectations that apply generically (to all ports). It would be nice to have something like LayoutTests/platform/generic/TestExpectations to which one could add new tests that are known to fail everywhere (e.g., because the code that implements a feature that is tested by those tests is not yet committed), but which will (at some point in the near future) not fail (when the code that is to be tested is committed). At present, it seems that if one wishes to do this, then it is necessary to add entries to the each base port expectations (i.e., chromium, mac, win, etc), which is rather annoying. If there is no objection to adding such a generic platform expectations file, then I will undertake to do so. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 11:02 AM, Glenn Adams gl...@skynav.com wrote: That would seem to work only if the test(s) fail the same way on all ports. In the case that I'm working from, I'm using reftests, where I know the correct expectations, but the actual behavior will (does) differ on different ports (when the corresponding feature is committed). That seems to indicate that ref test is not a good testing method for this feature. I would like to be able to (independently) commit new reftests *and* their known good expectation counterparts (that should apply on all ports), and then subsequently commit an independent patch that implements the expected behavior (on some but not all ports), and the comment further follow-on patches that fix the failing ports (possibly by writing new expectations for those specific ports). What kind of tests are you trying to add? Assuming this is related to your line break work, it might be possible to convert your tests to dumpAsText tests using getComputedStyle and check in failing *-expected.txt files. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke dpra...@chromium.org wrote: We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Hmm, I actually used port specific reftest expectation files in a recent patch [1] (since rolled out), and it appeared to work (as a way to effectively rebaseline those expectations). So something seems to be working. [1] http://trac.webkit.org/changeset/133529 At any rate, we encourage people these days to check in expected failures rather than suppressing them using the TestExpectations files. The problem is essentially a chicken and egg problem. I don't know what the per-port failures will be ahead of time, but I do know the set of correct expectations. Since I am (independently) unable to build/test all ports run by build bots, I would like to commit the set of tests plus known good expectations as a preliminary step (with a generic skip all tests for all ports), and then subsequently commit the feature itself, and then subsequently override the generic skip on a port specific basis, effectively re-enabling the tests on a port by port basis as I refine the feature patch (as needed) to handle port specific behavioral differences. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams gl...@skynav.com wrote: On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke dpra...@chromium.org wrote: We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Hmm, I actually used port specific reftest expectation files in a recent patch [1] (since rolled out), and it appeared to work (as a way to effectively rebaseline those expectations). So something seems to be working. [1] http://trac.webkit.org/changeset/133529 I expect it'll sort of work, but it won't be robust; you may hit weird behavior and/or bugs. We really haven't beaten on this aspect of things, and I don't know yet how much we want to. At any rate, we encourage people these days to check in expected failures rather than suppressing them using the TestExpectations files. The problem is essentially a chicken and egg problem. I don't know what the per-port failures will be ahead of time, but I do know the set of correct expectations. Since I am (independently) unable to build/test all ports run by build bots, I would like to commit the set of tests plus known good expectations as a preliminary step (with a generic skip all tests for all ports), and then subsequently commit the feature itself, and then subsequently override the generic skip on a port specific basis, effectively re-enabling the tests on a port by port basis as I refine the feature patch (as needed) to handle port specific behavioral differences. I think this is a reasonable approach. I would be interested to hear if others had alternatives they preferred. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
If we do add base test expectations shared by all platforms, please don’t put the file into LayoutTests/platform/generic; just put it at the top level of LayoutTests. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 11:35 AM, Dirk Pranke dpra...@chromium.org wrote: On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams gl...@skynav.com wrote: On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke dpra...@chromium.org wrote: We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Hmm, I actually used port specific reftest expectation files in a recent patch [1] (since rolled out), and it appeared to work (as a way to effectively rebaseline those expectations). So something seems to be working. [1] http://trac.webkit.org/changeset/133529 I expect it'll sort of work, but it won't be robust; you may hit weird behavior and/or bugs. We really haven't beaten on this aspect of things, and I don't know yet how much we want to. I don't think we should support port specific ref test results. That kind of misses the point of using a ref test in the first place. I mean, you may as well check in port specific pixel results which are easier to review for correctness. It may be the case that a ref test is not appropriate for what you're trying to test. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 12:04 PM, Darin Adler da...@apple.com wrote: On Nov 13, 2012, at 12:02 PM, Tony Chang t...@chromium.org wrote: I don't think we should support port specific ref test results. That kind of misses the point of using a ref test in the first place. I mean, you may as well check in port specific pixel results which are easier to review for correctness. I don’t agree that pixel results are easier to review for correctness. Here is a ref test result from ietestcenter: http://trac.webkit.org/browser/trunk/LayoutTests/ietestcenter/css3/flexbox/flexbox-flex-002-expected.htm Looking at that HTML file, it's not immediately obvious that the result is correct. If I had a png file, it would easy to see if there's red showing or not. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 12:02 PM, Tony Chang t...@chromium.org wrote: On Tue, Nov 13, 2012 at 11:35 AM, Dirk Pranke dpra...@chromium.org wrote: On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams gl...@skynav.com wrote: On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke dpra...@chromium.org wrote: We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Hmm, I actually used port specific reftest expectation files in a recent patch [1] (since rolled out), and it appeared to work (as a way to effectively rebaseline those expectations). So something seems to be working. [1] http://trac.webkit.org/changeset/133529 I expect it'll sort of work, but it won't be robust; you may hit weird behavior and/or bugs. We really haven't beaten on this aspect of things, and I don't know yet how much we want to. I don't think we should support port specific ref test results. That kind of misses the point of using a ref test in the first place. I mean, you may as well check in port specific pixel results which are easier to review for correctness. It may be the case that a ref test is not appropriate for what you're trying to test. I think that there are probably cases where we will have differences in results because of (legal and entirely correct or permissible) differences in the implementations and in this case a reftest can still be an improvement over maintaining N platform-specific pixel versions. The obvious example is when we haven't implemented features yet (or have bugs). The W3C's spec for handling reftests also gives you a way to say a test passes if any of these N references may match, which is fairly consistent with the idea that platform specific references are okay in some cases. As to whether pixel-tests are easier to review for correctness than reftests or not, I think it depends on the test. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
