Hi James,
On 21/09/2014 15:00 , James Graham wrote:
Obviously I agree that good testing of implementations is key to
interoperability. I also agree that we should encourage vendors to
create and run shared tests for the web technologies that we implement.
I am substantially less convinced that tying these tests to the spec
lifecycle makes sense. The W3C Process encourages people to think of
interoperability as a binary condition; either implementations are
interoperable or they are not. This leads to ideas like the CSS WG's
rule that two implementations must pass every test. On the face of it
this may appear eminently sensible. However the incentives for testing
under such a regime are not well aligned with the goal of finding bugs
in implementations; in essence people are encouraged to write as many
tests as are needed to satisfy the letter of the rules but to make them
all as shallow and unlikely to find bugs as possible to avoid causing
unwanted holdups in the specification process. I would much prefer to
have a testsuite written by people making a genuine effort to find
errors in implementations even if the result is that no one passes every
single test.
I couldn't agree more. In fact, part of my hope when we were setting up
Web Platform Tests was that having a continuously updated test suite
instead of a bunch of hasty rushes to get enough coverage of a spec for
it to "ship" would help people realise that specs should be handled in a
similar manner too.
I can't say it has brought about a revolution yet, but it has certainly
helped change minds. It's hard to argue against a continuously updated
test suite. It's hard to imagine that such an animal wouldn't find spec
bugs in addition to implementation bugs. It's hard to justify knowing
about bugs and not shipping an update. Things tend to make their own way
from there.
My concrete suggestion is that we, as an organisation, work to achieve
parity between the tests we require to ship our own code and those we
release in ways that can be used to support a spec and, more
importantly, those that can be used verify that different
implementations match up. In practice this means not writing tests in
Mozilla-specific formats, and making sure that we have a way to upstream
tests that we've written. Making this process as low-overhead as
possible is something that I'm working on.
A very strong +1.
However if we as an organisation really care about testing
core platform features which already have an implementation in gecko,
one way to achieve that would be to give someone working on Servo the
explicit job of creating testsuites for the big-ticket features they
implement as they implement them.
That would certainly be helpful.
In addition, I would note that while a shared test suite benefits
everyone. At this point Mozilla has proven to be a huge contributor to
the WPT project (with Opera's massive release of tests another notable
help) but we have not yet seen comparable commitments from the other
browser vendors. So any help you can provide in convincing people to
contribute is very welcome.
Maybe it all doesn't matter too much as long as implementors keep
reading the whatwg spec instead.
In terms of HTML at the W3C it's pretty clear that they've dropped the
ball, and haven't even realised it yet. There was a thread started five
days ago about the future of HTML after this release and so far it has
gained exactly no responses from implementors. The WG turned into an
unproductive, bureaucratic, nightmare and succeeded in driving out their
core constituency leaving the remaining participants to debate topics of
little consequence.
If we were to complain about the lack of testing for HTML, we would be
told in no uncertain terms that we (or at least the WG) had agreed to
the "Plan 2014" which explicitly chose to forego testing in certain
areas, and specification accuracy, in favour of shipping at a defined
time. This idea of shipping on a date-based schedule isn't actually
obviously bad, as long as you set the expectations correctly, which is
something the W3C will get wrong. It would indeed be nice if W3C would
embrace this fully and move to a WHATWG-style model with no eratta but a
spec kept continually up-to-date in the areas where there is a need for
change, and absolute rigidity in the areas where reality dictates that
change is impossible. However moving them there is a longer term goal
that seems to be met with extreme resistance from people who haven't
fully grasped how shipping a web browser works.
Well, my plan is to move pretty much there in a matter of months. For me
that's the biggest value in putting HTML5 out of the door: it frees up a
lot of flexibility (and energy) in how things are done from now on.
Constructive input is certainly welcome :)
--
Robin Berjon - http://berjon.com/ - @robinberjon
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
