On 1/8/2016 6:02 PM, James Graham wrote:
On 08/01/16 22:41, Robert O'Callahan wrote:
On Sat, Jan 9, 2016 at 10:27 AM, Benjamin Smedberg
<benja...@smedbergs.us>
wrote:
What are the implications of this?
The web-platform tests are pass/fail, right? So is it a bug if they
pass
but have different behaviors in e10s and non-e10s mode?
Yeah, I'm confused.
If a wpt test passes but with different output, then either there is no
problem or the test is incomplete and should be changed.
Maybe I should clarify.
web-platform-tests are slightly different to most tests in that we run
both tests we currently pass and tests that we currently don't pass.
On treeherder all we check is that we got the same result in this run
as we expected on the basis of previous runs.
Is this "same as previous run" behavior automatic, or manually
annotated? Running tests which don't pass is supported and common on
many other test suites: fail-if and random-if are used to mark tests as
a known fail but still run them.
Is this a temporary state, where the end goal is to have the
web-platform tests use similar manifests to all of our other tests? Can
you provide some context about why web-platform tests are using
different infrastructure from everywhere else?
The effect of all of this is that in order to understand what's
actually needed to bring e10s builds up to par with non-e10s builds
you need to look at the actual test results rather than just the list
of disabled tests. I believe that there are both instances of tests
that pass in non-e10s but not in e10s builds, and the reverse. wptview
gives you the ability to do that using data directly from treeherder.
The actual action to take on the basis of this data is obviously
something for the people working on e10s to determine.
This is not the responsibility of the e10s team; this is an all-hands
effort as we switch to making e10s the default configuration and soon
the only configuration for Firefox. If having different results for e10s
and non-e10s is not expected, who is the module owner/responsible for
the web platform tests and can create a list of the problem results and
find owners to get each one fixed?
--BDS
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
