On 04/11/15 10:24, Robert O'Callahan wrote:
On Wed, Nov 4, 2015 at 5:14 PM, James Graham <ja...@hoppipolla.co.uk> wrote:
On 03/11/15 22:08, Robert O'Callahan wrote:
Why not create a Mochitest compatibility layer over testharness.js so
that tests that only use SimpleTest.waitForExplicitFinish(),
SimpleTest.finish(), is(), todo() and ok() can run on Servo with trivial
changes?
A few reasons I am skeptical of this:
* Mochitests test gecko behaviour, not standardised behaviour. These can
and do differ. Therefore they are even less trustworthy than the average
test.
I'm not sure what you mean by this. It's easy to write web-platform-tests
that depend on non-standard behavior. I had to debug one such last week.
Furthermore Gecko mochitests have been tested against Gecko, which has
itself been tested on the Web, so Gecko mochitests are far more likely to
be "correct" than a test someone just wrote and only tested on, say, Servo.
I mean that mochitests are used as regression tests; they aren't
necessarily testing the per-spec behaviour (which might match Blink or
Edge, but not Gecko), but whatever the Gecko code implements. Of course
web-platform-tests can be buggy like any other code, but they are at
least intended to test the right thing.
* Mochitests are really written assuming the full gecko featureset. Given
the total lack of isolation between asserts, they may behave rather
unexpectedly and misleadingly in a browser not having all those features.
Many use Gecko-specific features but many don't.
Not just use of gecko-specific (or, often spidermonkey-specific)
behaviours, but also the fact that mochitests assume an assert never
fails (because if one does it's a regression that needs to be fixed). In
Servo, which might not implement all the features used, an assert can be
expected to fail for some time until more feature work is done. This is
particularly true at the moment when Servo doesn't implement the window
error handler. I anticipate this impedance mismatch would cause the
tests to be less useful in Servo than they are in Gecko.
* In the long term having multiple APIs for writing tests that people have
to learn in order to read tests is a big net negative. If we allow
mochitests to be upstreamed with a shim we should expect other vendors to
do the same, and to end up with half a dozen possible test formats.
That's fair, though it's a problem we introduced when we started adding
web-platform-tests to the tree.
Put it this way: when Servo needs tests for a feature, I expect it makes
much more sense to take Gecko mochitests for the feature and convert them
to web-platform-tests than to write new tests from scratch.
Well sure, I agree that taking mochitests as the input to a test-writing
effort is a good idea. I see this as being very different to blindly
shimming mochitests into the wpt harness. Having said that, however, I
don't think people have complained a lot about lack of test coverage
from wpt except in the areas that it doesn't cover at all i.e. dynamic
changes to layout, or other human interaction.
_______________________________________________
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo
