Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[Robert O'Callahan]

On Thu, Nov 5, 2015 at 12:44 AM, James Graham 
wrote:

> On 04/11/15 11:41, Robert O'Callahan wrote:
>
>> The media tests are better than I thought --- I found more. They don't
>> test
>> the variety of problematic media files that our mochitests do, but maybe
>> that's not in scope? Should we be worried that regressions in the media
>> stack seldom trigger W test failures on mozilla-inbound but often trigger
>> mochitest failures?
>>
>
> I consider anything that might affect browser interop or quality in-scope,
> as long as it can be run cross-browser and it's clear what the expected
> result is.


OK: https://github.com/w3c/web-platform-tests/issues/2305

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Handling cases where the spec is inefficient and not followed by anyone

[Manish Goregaokar]

Oh, yeah, in that case we should just mimic the other browsers and file
bugs.

-Manish Goregaokar
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] loading web fonts synchronously during automation

[Bobby Holley]

Seems generally doable, but probably more than I want to chew right now.
I'll get an issue filed though.

Does anyone object to landing the synchronous font loading as a stopgap to
get better reftest coverage until we make the font loading logic smarter?

On Wed, Nov 4, 2015 at 8:14 PM, Bobby Holley  wrote:

> That's a great point - I'll take a look to see how doable it is.
>
>
> On Wed, Nov 4, 2015 at 7:28 PM, L. David Baron  wrote:
>
>> On Wednesday 2015-11-04 19:16 -0800, Bobby Holley wrote:
>> > Right now, every wpt test includes a stylesheet with an @font-face rule
>> to
>> > make the 'ahem' font available to tests. Since font loading triggers a
>> > document-wide reflow, we end up with a non-deterministic network-driven
>> > reflow in each and every test, which makes it harder to test layout
>> > optimizations and increases the chances for intermittent bugs.
>> >
>> > In [1] I'm planning to add a debug flag to load web fonts synchronously,
>> > and set that flag in the wpt runner. Please let me know if anyone has
>> any
>> > objections.
>>
>> The font shouldn't even load unless it's needed (i.e., there's text
>> that uses it).  I *think* that's the way things work across
>> browsers, and allows inclusion of style sheets that provide a
>> "library" of @font-face rules that may or may not be used.  That
>> would avoid the reflow except where Ahem is actually used, which
>> seems like the way things ought to work.
>>
>> -David
>>
>> --
>> 턞   L. David Baron http://dbaron.org/   턂
>> 턢   Mozilla  https://www.mozilla.org/   턂
>>  Before I built a wall I'd ask to know
>>  What I was walling in or walling out,
>>  And to whom I was like to give offense.
>>- Robert Frost, Mending Wall (1914)
>>
>
>
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] loading web fonts synchronously during automation

[Bobby Holley]

That's a great point - I'll take a look to see how doable it is.

On Wed, Nov 4, 2015 at 7:28 PM, L. David Baron  wrote:

> On Wednesday 2015-11-04 19:16 -0800, Bobby Holley wrote:
> > Right now, every wpt test includes a stylesheet with an @font-face rule
> to
> > make the 'ahem' font available to tests. Since font loading triggers a
> > document-wide reflow, we end up with a non-deterministic network-driven
> > reflow in each and every test, which makes it harder to test layout
> > optimizations and increases the chances for intermittent bugs.
> >
> > In [1] I'm planning to add a debug flag to load web fonts synchronously,
> > and set that flag in the wpt runner. Please let me know if anyone has any
> > objections.
>
> The font shouldn't even load unless it's needed (i.e., there's text
> that uses it).  I *think* that's the way things work across
> browsers, and allows inclusion of style sheets that provide a
> "library" of @font-face rules that may or may not be used.  That
> would avoid the reflow except where Ahem is actually used, which
> seems like the way things ought to work.
>
> -David
>
> --
> 턞   L. David Baron http://dbaron.org/   턂
> 턢   Mozilla  https://www.mozilla.org/   턂
>  Before I built a wall I'd ask to know
>  What I was walling in or walling out,
>  And to whom I was like to give offense.
>- Robert Frost, Mending Wall (1914)
>
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] loading web fonts synchronously during automation

[L. David Baron]

On Wednesday 2015-11-04 19:16 -0800, Bobby Holley wrote:
> Right now, every wpt test includes a stylesheet with an @font-face rule to
> make the 'ahem' font available to tests. Since font loading triggers a
> document-wide reflow, we end up with a non-deterministic network-driven
> reflow in each and every test, which makes it harder to test layout
> optimizations and increases the chances for intermittent bugs.
> 
> In [1] I'm planning to add a debug flag to load web fonts synchronously,
> and set that flag in the wpt runner. Please let me know if anyone has any
> objections.

The font shouldn't even load unless it's needed (i.e., there's text
that uses it).  I *think* that's the way things work across
browsers, and allows inclusion of style sheets that provide a
"library" of @font-face rules that may or may not be used.  That
would avoid the reflow except where Ahem is actually used, which
seems like the way things ought to work.

-David

-- 
턞   L. David Baron http://dbaron.org/   턂
턢   Mozilla  https://www.mozilla.org/   턂
 Before I built a wall I'd ask to know
 What I was walling in or walling out,
 And to whom I was like to give offense.
   - Robert Frost, Mending Wall (1914)


signature.asc
Description: Digital signature
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Handling cases where the spec is inefficient and not followed by anyone

[Boris Zbarsky]

On 11/4/15 12:09 PM, Manish Goregaokar wrote:

I'd prefer to avoid diverging from the spec wherever possible -- even if
other browsers disagree (with the spec and with each other)


I think Ms2ger was talking about cases in which the spec disagrees with 
all browsers, who agree with each other.


In which case, chances are the spec is just wrong.  Unless it has an 
explicit note about how it's purposefully disagreeing with the world 
because it thinks it's better that way.  In which case it may still be 
wrong, of course.


Note that there are obvious and fundamental bugs in specs all the time. 
 Pretty much any time I look at a spec for anything at all complicated 
I find a few.  :(


-Boris
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[Robert O'Callahan]

On Thu, Nov 5, 2015 at 12:17 AM, James Graham 
wrote:

> On 04/11/15 11:12, Robert O'Callahan wrote:
>
>> Well sure, I agree that taking mochitests as the input to a test-writing
>>> effort is a good idea. I see this as being very different to blindly
>>> shimming mochitests into the wpt harness. Having said that, however, I
>>> don't think people have complained a lot about lack of test coverage from
>>> wpt except in the areas that it doesn't cover at all i.e. dynamic changes
>>> to layout, or other human interaction.
>>>
>>
>>
>> To pick a couple of areas I work on: for CSSOM-Views for example there is
>> practically no coverage at all. The media tests are better but still very
>> limited compared to what we test in mochitests.
>>
>
> It would be great to have a record of some areas where we know that
> web-platform-tests has missing coverage; we occasionally get people looking
> for areas where they can make useful test contributions and asking what's
> required. Can you file some issues on GitHub, possibly pointing to the
> relevant mochitests that we could draw from?


Sure. https://github.com/w3c/web-platform-tests/issues/2304

The media tests are better than I thought --- I found more. They don't test
the variety of problematic media files that our mochitests do, but maybe
that's not in scope? Should we be worried that regressions in the media
stack seldom trigger W test failures on mozilla-inbound but often trigger
mochitest failures?

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[James Graham]

On 04/11/15 11:41, Robert O'Callahan wrote:


Sure. https://github.com/w3c/web-platform-tests/issues/2304


Thanks!


The media tests are better than I thought --- I found more. They don't test
the variety of problematic media files that our mochitests do, but maybe
that's not in scope? Should we be worried that regressions in the media
stack seldom trigger W test failures on mozilla-inbound but often trigger
mochitest failures?


I consider anything that might affect browser interop or quality 
in-scope, as long as it can be run cross-browser and it's clear what the 
expected result is.

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Handling cases where the spec is inefficient and not followed by anyone

[Ms2ger]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 11/02/2015 08:21 PM, Manish Goregaokar wrote:
> We've recently had two 
> cases  where:
> 
> - The spec suggests something which if implemented correctly, would
> lead to inefficiencies - Major browsers do not follow that portion
> of the spec and usually have a faster/cleaner implementation which
> isn't spec-conformant. They usually don't match on how it's done,
> either. - Spec bugs on the issue haven't really gotten anywhere
> 
> We have three choices here, we can wait indefinitely until the spec
> gets fixed, we can implement the spec as is (which require major
> changes and affect perf or complexity), or we can hope that nobody
> relies on this behaviour (given that it's not followed by major
> browsers) and implement it as logically as possible, keeping in
> line with other browsers (and leaving a bug open about the spec
> issue).
> 

I'm not sure why you scoped this to inefficient spec algorithms. In
any case where the specifications is wrong (in particular, where all
browsers consistently disagree with the specification, and no browser
vendor wants to match the specification in the (possibly distant)
future), we should submit that feedback, write tests, and implement
the interoperable behaviour.

HTH
Ms2ger

-BEGIN PGP SIGNATURE-

iQEcBAEBAgAGBQJWOiiDAAoJEOXgvIL+s8n2lgQH/1Zns5Mdc8jtnyxrldC+Luh4
BTJJoLjvtfPQZp/NA7Q8E1Xy3pCRrguJTWQCZHSx0g1+vsH2/fclr0JsF3hRQ4bS
ztE3Ww/mWJDQVk/eKTesYyKqExOxLT1bkUajeIov4SNBrke6kR7kAjJOzx2cAaD+
g8kf+rXo5a1HmQprsKRuWrpLdYsZxDHBXW+uGchyqB1TO5Wc5Pni1dfXvC5zDG+y
UvtHpeLBz3MrCPhYoCFtLB/d7x6/jnP9+UsBBl9YY7DGW1TmtcXIlC1rexYSgzGU
E//G0l9+D4p6YhOsAr6QhquZUXvkyusvH/gbhqJ9fhXrZMWfI9jxaq4syazoK3A=
=A0uJ
-END PGP SIGNATURE-
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Handling cases where the spec is inefficient and not followed by anyone

[Manish Goregaokar]

On Wed, Nov 4, 2015 at 9:17 PM, Ms2ger  wrote:

> I'm not sure why you scoped this to inefficient spec algorithms


Because inefficient or complexity-inducing specs cause problems, the rest,
less so.

If the spec isn't consistently implemented across the board, but
implementing the spec is low-effort (and doesn't hit perf) for us,
we can just implement it, file a spec bug, and reimplement if it ever gets
fixed. Whereas if it's something like the :target or
HTMLFormControlsCollection thing,we have to needlessly implement extra
functionality or refactor parts of DOM introducing inefficiencies,
and if the spec ever gets fixed, rip it all out.

I'd prefer to avoid diverging from the spec wherever possible -- even if
other browsers disagree (with the spec and with each other), implementing
what the spec says (rather than picking our favorite browser and mimicing
it) and filing a bug -- if it's not too hard to do so -- is what we should
do.

-Manish Goregaokar
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[Robert O'Callahan]

On Wed, Nov 4, 2015 at 5:14 PM, James Graham  wrote:

> On 03/11/15 22:08, Robert O'Callahan wrote:
>
> Why not create a Mochitest compatibility layer over testharness.js so
>> that tests that only use SimpleTest.waitForExplicitFinish(),
>> SimpleTest.finish(), is(), todo() and ok() can run on Servo with trivial
>> changes?
>>
>
> A few reasons I am skeptical of this:
>
> * Mochitests test gecko behaviour, not standardised behaviour. These can
> and do differ. Therefore they are even less trustworthy than the average
> test.
>

I'm not sure what you mean by this. It's easy to write web-platform-tests
that depend on non-standard behavior. I had to debug one such last week.
Furthermore Gecko mochitests have been tested against Gecko, which has
itself been tested on the Web, so Gecko mochitests are far more likely to
be "correct" than a test someone just wrote and only tested on, say, Servo.

* Mochitests are really written assuming the full gecko featureset. Given
> the total lack of isolation between asserts, they may behave rather
> unexpectedly and misleadingly in a browser not having all those features.
>

Many use Gecko-specific features but many don't.

* In the long term having multiple APIs for writing tests that people have
> to learn in order to read tests is a big net negative. If we allow
> mochitests to be upstreamed with a shim we should expect other vendors to
> do the same, and to end up with half a dozen possible test formats.
>

That's fair, though it's a problem we introduced when we started adding
web-platform-tests to the tree.

Put it this way: when Servo needs tests for a feature, I expect it makes
much more sense to take Gecko mochitests for the feature and convert them
to web-platform-tests than to write new tests from scratch.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[James Graham]

On 04/11/15 10:24, Robert O'Callahan wrote:

On Wed, Nov 4, 2015 at 5:14 PM, James Graham  wrote:


On 03/11/15 22:08, Robert O'Callahan wrote:

Why not create a Mochitest compatibility layer over testharness.js so

that tests that only use SimpleTest.waitForExplicitFinish(),
SimpleTest.finish(), is(), todo() and ok() can run on Servo with trivial
changes?



A few reasons I am skeptical of this:

* Mochitests test gecko behaviour, not standardised behaviour. These can
and do differ. Therefore they are even less trustworthy than the average
test.



I'm not sure what you mean by this. It's easy to write web-platform-tests
that depend on non-standard behavior. I had to debug one such last week.
Furthermore Gecko mochitests have been tested against Gecko, which has
itself been tested on the Web, so Gecko mochitests are far more likely to
be "correct" than a test someone just wrote and only tested on, say, Servo.


I mean that mochitests are used as regression tests; they aren't 
necessarily testing the per-spec behaviour (which might match Blink or 
Edge, but not Gecko), but whatever the Gecko code implements. Of course 
web-platform-tests can be buggy like any other code, but they are at 
least intended to test the right thing.



* Mochitests are really written assuming the full gecko featureset. Given

the total lack of isolation between asserts, they may behave rather
unexpectedly and misleadingly in a browser not having all those features.



Many use Gecko-specific features but many don't.


Not just use of gecko-specific (or, often spidermonkey-specific) 
behaviours, but also the fact that mochitests assume an assert never 
fails (because if one does it's a regression that needs to be fixed). In 
Servo, which might not implement all the features used, an assert can be 
expected to fail for some time until more feature work is done. This is 
particularly true at the moment when Servo doesn't implement the window 
error handler. I anticipate this impedance mismatch would cause the 
tests to be less useful in Servo than they are in Gecko.



* In the long term having multiple APIs for writing tests that people have

to learn in order to read tests is a big net negative. If we allow
mochitests to be upstreamed with a shim we should expect other vendors to
do the same, and to end up with half a dozen possible test formats.



That's fair, though it's a problem we introduced when we started adding
web-platform-tests to the tree.

Put it this way: when Servo needs tests for a feature, I expect it makes
much more sense to take Gecko mochitests for the feature and convert them
to web-platform-tests than to write new tests from scratch.


Well sure, I agree that taking mochitests as the input to a test-writing 
effort is a good idea. I see this as being very different to blindly 
shimming mochitests into the wpt harness. Having said that, however, I 
don't think people have complained a lot about lack of test coverage 
from wpt except in the areas that it doesn't cover at all i.e. dynamic 
changes to layout, or other human interaction.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[Robert O'Callahan]

On Wed, Nov 4, 2015 at 11:50 PM, James Graham 
wrote:

> On 04/11/15 10:24, Robert O'Callahan wrote:
>
>> On Wed, Nov 4, 2015 at 5:14 PM, James Graham 
>> wrote:
>>
>> On 03/11/15 22:08, Robert O'Callahan wrote:
>>>
>>> Why not create a Mochitest compatibility layer over testharness.js so
>>>
 that tests that only use SimpleTest.waitForExplicitFinish(),
 SimpleTest.finish(), is(), todo() and ok() can run on Servo with trivial
 changes?


>>> A few reasons I am skeptical of this:
>>>
>>> * Mochitests test gecko behaviour, not standardised behaviour. These can
>>> and do differ. Therefore they are even less trustworthy than the average
>>> test.
>>>
>>>
>> I'm not sure what you mean by this. It's easy to write web-platform-tests
>> that depend on non-standard behavior. I had to debug one such last week.
>> Furthermore Gecko mochitests have been tested against Gecko, which has
>> itself been tested on the Web, so Gecko mochitests are far more likely to
>> be "correct" than a test someone just wrote and only tested on, say,
>> Servo.
>>
>
> I mean that mochitests are used as regression tests; they aren't
> necessarily testing the per-spec behaviour (which might match Blink or
> Edge, but not Gecko), but whatever the Gecko code implements. Of course
> web-platform-tests can be buggy like any other code, but they are at least
> intended to test the right thing.


I think it's uncommon for mochitests to intentionally be stricter than what
the spec requires ... excluding cases where the spec is known to be
inadequate. It's easy to be accidentally more strict, but that's true for
WPT too.


> * Mochitests are really written assuming the full gecko featureset. Given
>>
>>> the total lack of isolation between asserts, they may behave rather
>>> unexpectedly and misleadingly in a browser not having all those features.
>>>
>>>
>> Many use Gecko-specific features but many don't.
>>
>
> Not just use of gecko-specific (or, often spidermonkey-specific)
> behaviours, but also the fact that mochitests assume an assert never fails
> (because if one does it's a regression that needs to be fixed). In Servo,
> which might not implement all the features used, an assert can be expected
> to fail for some time until more feature work is done. This is particularly
> true at the moment when Servo doesn't implement the window error handler. I
> anticipate this impedance mismatch would cause the tests to be less useful
> in Servo than they are in Gecko.
>

That makes sense.


>
> * In the long term having multiple APIs for writing tests that people have
>>
>>> to learn in order to read tests is a big net negative. If we allow
>>> mochitests to be upstreamed with a shim we should expect other vendors to
>>> do the same, and to end up with half a dozen possible test formats.
>>>
>>>
>> That's fair, though it's a problem we introduced when we started adding
>> web-platform-tests to the tree.
>>
>> Put it this way: when Servo needs tests for a feature, I expect it makes
>> much more sense to take Gecko mochitests for the feature and convert them
>> to web-platform-tests than to write new tests from scratch.
>>
>
> Well sure, I agree that taking mochitests as the input to a test-writing
> effort is a good idea. I see this as being very different to blindly
> shimming mochitests into the wpt harness. Having said that, however, I
> don't think people have complained a lot about lack of test coverage from
> wpt except in the areas that it doesn't cover at all i.e. dynamic changes
> to layout, or other human interaction.


To pick a couple of areas I work on: for CSSOM-Views for example there is
practically no coverage at all. The media tests are better but still very
limited compared to what we test in mochitests.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[James Graham]

On 04/11/15 11:12, Robert O'Callahan wrote:

Well sure, I agree that taking mochitests as the input to a test-writing
effort is a good idea. I see this as being very different to blindly
shimming mochitests into the wpt harness. Having said that, however, I
don't think people have complained a lot about lack of test coverage from
wpt except in the areas that it doesn't cover at all i.e. dynamic changes
to layout, or other human interaction.



To pick a couple of areas I work on: for CSSOM-Views for example there is
practically no coverage at all. The media tests are better but still very
limited compared to what we test in mochitests.


It would be great to have a record of some areas where we know that 
web-platform-tests has missing coverage; we occasionally get people 
looking for areas where they can make useful test contributions and 
asking what's required. Can you file some issues on GitHub, possibly 
pointing to the relevant mochitests that we could draw from?


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo