On 08/16/2012 05:10 PM, Ehsan Akhgari wrote:
I think this is generally a good idea. I have a few questions before
jumping in to agree though.
1. Is the current testharness.js API the documentation at the beginning
of <http://w3c-test.org/resources/testharness.js>? If that is the case,
the API looks a lot heavier weight than the default mochitest API we
use. In that case, would it make sense for us to have a compatibility
layer which translates our mochitest APIs to the equivalent
testharness.js API calls? (I'm not 100% sure if that conversion would
be straightforward.)
I don't feel it's terribly heavyweight. In the simple case, you need a
test(function() {
// tests
});
That's two extra lines of boilerplate. Our mochitest boilerplate is…
(*looks it up*) 30 lines. I think we'll deal. The method names for
assertions (assert_*) are a bit longer than we're used to, but in my
experience, you get used to it rather quickly.
IMO, it would not be a good idea to try to layer another API on top of
this, because it makes our tests harder to understand and less reusable
by other browser vendors, and it means that experience with our brand of
testharness.js doesn't help much to understand the standard brand.
I also don't think it would be terribly straightforward, and I think
that having real mochitests, testharness.js-that-look-like-mochitest and
standard testharness.js in our tree would make our testing situation
more confusing than the lessened learning curve would be worth. In
particular, having two APIs that superficially look the same but are
actually built on unrelated frameworks seems a recipe for a lot of
annoying differences in edge cases.
2. Is there any support for running reftest-style tests in a framework
that is reusable by other browsers? If not, can we move to propose the
reftest framework to the appropriate standards bodies so that it can be
adopted by other browsers? Our reftest framework has been carefully
designed to be Gecko-agnostic, and is much superior to the equivalent
testing framework that WebKit has (not sure about other browser
engines). Furthermore, the files loaded by this framework are not
loaded in a privileged context with APIs such as SpecialPowers, which
makes a large number of them portable to other browser engines.
The W3C has already adopted reftests. See
<https://test.csswg.org/harness/test/CSS-STYLE-ATTR_DEV/single/style-attr-braces-001/>
for example.
In fact, we've already got a place in m-c to put reftests to be
submitted to the CSSWG:
<http://mxr.mozilla.org/mozilla-central/source/layout/reftests/w3c-css/submitted/README?force=1>.
The main issue here is that the CSSWG appears to impose rather stringent
test formatting requirements, which makes writing reftests for them much
more of a drag than just writing them for ourselves.
I think it makes sense for us if we can start this effort on the reftest
framework, since that has a much lower barrier to entry, and ultimately
this effort would be valuable only if other browser engines start to use
our tests (and hopefully share theirs with us as well).
Opera already shares a lot of their tests, and Google, Apple and
Microsoft have been known to submit tests as well, some of which are
already running on tinderbox.
HTH
Ms2ger
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
