Re: [dev-servo] W3C CSS Test Suites

[James Graham]

On 02/09/13 14:40, Simon Sapin wrote:

Le 02/09/2013 13:04, Sanghyeon Seo a écrit :

Servo can now run some of tests in Official W3C CSS Test Suites.
http://www.w3.org/Style/CSS/Test/

For example, clear-001, 002, 003, 005 pass, and clear-004 fails
because of
paint order (#677).

I would like to integrate passing tests to Servo. What would be the
best way
to do so?


Unfortunately (at least for the CSS 2.1 test suite), very few of these
tests are reftests that can be automated. Most of them require human
confirmation, which is just not viable IMO.

I think it is more valuable to spend time on importing reftests, both
from W3C and Gecko.



In general the best source of cross-browser tests is [1] (for CSS) and 
[2] (for the rest of the web platform, excluding JS). These are designed 
to work without special magic browser-internal APIs and as such should 
work on servo as soon as it supports the relevant features.


The testing work for servo should be focused on getting the automatable 
parts of those testsuites running. This includes anything in 
web-platform-tests not marked -manual (or at least will once I fix up 
the whole repository to match the convention) and almost all of the css3 
tests in csswg-test. Actually automating the tests may require some work 
on the ability to drive servo externally. For example if one can inject 
script from an external process and return data to that process it is 
sufficient to run all the javascript-based tests. Alternatively, the 
same goal could be met by a way to load urls and a special API for 
writing to stdout. If one can navigate to a particular URL and discover 
when that load finished (possibly by injecting script, as before), that 
is sufficient to run the reftests (one can always use an external 
screenshot API rather than cloning the gecko approach of using 
extensions to canvas, and indeed this has a number of advantages).


It should also be a goal that any new tests *written* for servo but not 
testing the servo internals end up in these repositories rather than in 
some servo-specific collection of tests. Fragmentation of testsuites is 
a major impediment to interoperability between browsers. The greenfield 
implementation of the platform in Servo provides a great opportunity to 
find holes in the parts where others are afraid to make changes. Turning 
this implementation experience into testcases is a unique opportunity to 
find and fix holes in the specs, and create an environment where 
everyone is confident that following the spec and passing the related 
tests will lead to web-compatible results.


[1] https://github.com/w3c/csswg-test
[2] https://github.com/w3c/web-platform-tests
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] What is the best practice to polish commits in 1 pull request?

[James Graham]

On 30/01/14 14:39, Tetsuharu OHZEKI wrote:

Hi everyone,

I sometimes break up the critic's review by using `git commit --amend`
to polish my commits. (I'm sorry)

I read the issue (https://github.com/mozilla/servo/issues/1468) to
learn how to polish commits, but I feel that it have not gotten the
consensus of the best practice to polish commits with your reviews.


I don't know what counts as consensus, but I will describe what I think 
best practice ought to be, and hopefully explain why.


When you use a critic-like review system which stores a record of which 
commits have been reviewed, and which issues have been addressed, 
continually squashing the review into a single "patch" in the way that 
you would with mq doesn't really work, because the review tool is using 
its record of the existing commits to track where comments have been 
made and what has already been reviewed.


Even in GitHub (without critic) this continual-squash mode of operation 
is a problem because each new set of changes to a patch cause all 
comments on previous version to be lost, so you essentially have to 
start the review from scratch on each new push.


Therefore I think the optimal system is one where you don't alter 
history at all until the review is accepted. At this point you rebase 
onto latest master and squash into a single commit. If that resulted in 
a non-trivial merge you force-push the branch and get review of the 
resulting "effective merge commit". Otherwise just force-push the branch 
and ask the reviewer to mark as r+ in github so that bors integrates a 
single commit.


So in summary, I think the best practice is :

* Don't do anything that requires a git push -f until your review is 
accepted (or unless you really know what you are doing)


* Once your review is accepted, squash into logical commits, rebase on 
master and git push -f


* If for some reason you need to rebase, and then make more commits, 
push the rebase first, then get the review rebased on critic (I can help 
with this, as can Ms2ger) before you push your new commits.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Roadmap Q2 goal : Pass Acid3

[James Graham]

On 25/02/14 11:42, David Bruant wrote:

Hi,

If I may weigh in, I wonder if passing Acid3 is a valuable goal in the
short term.
IE8 scores 20/100 at Acid3. All major websites and most websites have to
support IE8, because it's the current "boat-anchor browser" [1]. So most
websites don't need the features that make a 100 score.
Specifically, SVG might be a big piece and since IE8 doesn't support it,
usage is still very rare in websites.

Aiming at IE8 parity (... or rather the intersection of IE8 and current
standards, obviously) feels like a more practical goal if you want to
support the web as it is today. An incomplete list of relevant features
to get to IE8 parity [2].

Said differently, aiming at IE8 parity can only yield a much much better
supported-website-count/effort ratio than passing Acid3.


I think I agree that passing Acid 3 in the next four months is not going 
to happen. But that's more because it requires a whole set of 
technologies that we don't have an implementation of in Servo (SVG, DOM 
Range+traversal, etc.) and which don't obviously seem like high 
priorities compared to getting the fundamental architecture right.


I also think that the acid tests in general and acid 3 in particular are 
rather flawed tests; they make great PR pieces and that's a good reason 
to target them, but the actual things that they test are a grab-bag of 
random features and implementation bugs from the time they were written.


Having said that, I don't think that using "IE8 parity" as a metric is 
very useful. IE8 takes a lot of non-standard codepaths on real sites, so 
merely targeting the standard features it implements is not going to 
give you an end result that is actually compatible with all the sites 
that work with IE8 (for example, I believe that IE8 still has the old 
attachEvent junk rather than the standard addEventListener API, so by a 
literal reading of the above you wouldn't implement events at all).


I think a better way to proceed is to try to implement the parts that 
change the underlying architecture as early as possible (so, on the DOM 
side, this would be HTML parsing, page loading, script scheduling, 
history navigation, etc.) and as a minimum target as-good-as-gecko (or 
webkit or blink, or whatever) scores on the relevant web-platform-tests 
for those features (in practice we want to aim for 100% of course, but 
as long as we at least pass the tests that everyone else passes the 
compat. story should be OK). Once this is done we survey web content to 
see which higher-level features are most used and implement those in 
priority order using the same approach.


My method isn't likely to produce the best short-term gains, but I think 
it's the best route to compatibility with real sites in the long term.

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] HTML parsing alternatives

[James Graham]

On 06/03/14 02:05, Keegan McAllister wrote:


Writing our own HTML5 parser would be a lot of work, but does not
seem infeasible.  The parsers I've found (including the translated
C++ code for Gecko) are in the 10-20 KLoC range.  We can do a
one-time translation from Java for the most mechanical parts, without
building a complete translator.


FWIW I would estimate that a from-scratch implementation of a HTML 
parser that could replace Hubbub would be a "summer of code" sized 
project i.e. I would expect a reasonably new contributer to manage it in 
a couple of months and an experienced contributer to manage it in much 
less than that. Indeed much of hubbub itself was originally done as a 
GSoC project [1].



There is a standard test suite [2] for static HTML5 parsers.
Browsers have additional requirements due to speculation and
document.write(), but it looks like [3] Gecko implements that outside
the translated parser, so this is code we would have to write and
test in any case.


So part of the difficulty of document.write comes from the fact that it 
has to interact with the script loading / document lifecycle. Therefore 
it's going to be hard to get those parts of (any) parser right until we 
actually implement a more correct model of document loading. Ideally the 
two things would be designed concurrently so that there isn't an 
impedance mismatch between the parser and the loading code.



For the short term I will continue to work on the translator and see
if we can get more clarity about some of these unknowns.  But I'm
also inclined to try implementing parts of a new HTML5 parser in
Rust.  At any rate we should pay close attention to Gecko's parser
design, and I will continue reading through that code.


My suspicion is that it's possible to spend more time talking about 
various options than it would take to stand up a rough prototype parser 
(with e.g. less important tokenizer/treebuilder states missing). 
Therefore I think this sounds like a great idea.


[1] http://www.netsurf-browser.org/developers/gsoc/
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] HTML parsing alternatives

[James Graham]

On 07/03/14 07:13, Henri Sivonen wrote:


So part of the difficulty of document.write comes from the fact
that it has to interact with the script loading / document
lifecycle. Therefore it's going to be hard to get those parts of
(any) parser right until we actually implement a more correct
model of document loading. Ideally the two things would be
designed concurrently so that there isn't an impedance mismatch
between the parser and the loading code.


That makes sense.  What should I read to start understanding the
design requirements?


If I understand correctly what this is referring to, the short
version is that DOMContentLoaded interaction with 

Re: [dev-servo] character encoding in the HTML parser

[James Graham]

On 10/03/14 23:54, Keegan McAllister wrote:

Should we implement character encoding detection [1] at the same time
as the rest of the HTML parser?  It seems to be separable; the only
design interactions I see are:

- The character decoder and script APIs can write into the same input
stream - The encoding can change during parsing [2], which can be
handled as a mostly-normal navigation

Also, should we follow Gecko in representing the input stream as a
queue of UTF-16 buffers?  With UTF-8 we would have about half as much
data to stream through the parser, and (on 64-bit) we could do
case-insensitive ASCII operations 8 characters at a time.

Most content [3] is UTF-8, and the trend is in that direction.  But
for non-UTF-8 content we would pay the price of two conversions, if
the ultimate product is UCS-2 DOM strings.  However I don't think
we've finalized our representation of DOM strings, and we might do
the UCS-2 conversion lazily [4].


Apart from the compat problems that Simon already mentioned (which I 
think are important), it's worth considering in what cases the parser is 
likely to be a bottleneck. I believe that in load-type situations the 
parser is almost never the performance limiting factor (assuming a 
reasonably well optimised implementation); it's much more likely that 
network performance, layout, or scripting will dominate the time to 
load. On the other hand I think that there exist pages which do things 
like run innerHTML in performance-critical loops. Therefore I expect it 
makes more sense for the parser to operate on the same string type as 
the DOM than the same type as the network data.

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Web-platform-tests status

[James Graham]

On 19/04/14 15:32, Manish Goregaokar wrote:



* we don't have an easy way to run single tests


run.py (in src/test/wpt) has an --include option. Be sure to source
the virtualenv in the build dir before running run.py


FWIW if we had mach we could use the mach script which would 
automatically sort out the virtualenv for you.



* Servo times out on a lot of tests, causing the run to take 11
minutes here


We probably should exclude some of the tests in the default run


Well I think I need to have an option to take some kind of manifest file 
as input with rules to in/exclude certain directories. I need to 
implement that next for Firefox OS anyway, so it will be done early next 
week.



* reftests aren't run yet


Again, we haven't --included all of the tests yet, so this can be
simply by adding more directories there. test_types by default runs
both types of tests.


Yeah, but the servo support in wptrunner doesn't include reftest support 
yet. It's not hard to add or anything, it just hasn't been a priority.

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Web-platform-tests status

[James Graham]

On 19/04/14 16:24, Jack Moffitt wrote:

Thanks to the work of (in particular) jgraham and Manishearth, we can now
run web-platform-tests (the parts starting with /dom) in Servo with the
`make check-wpt` target.


Great work everyone! This will be a great source of good bugs for new
contributors I think and a great way to evaluate our progress against
other engines.


Yup. I have the idea of making "Are We Viable Yet" which will take the 
latest results from Gecko and Servo (and maybe later other browsers if 
they start running the tests on a regular basis) and display pass/fail 
for each test directory based on whether Servo passes at least the 
subset of tests passed by Gecko.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Fwd: Meeting notes 9/29 (CI; DOM binding safety; SpiderMonkey upgrade; test harness improvements; dogfooding; security; workweek; incremental reflow)

[James Graham]

On 30/09/14 14:40, Josh Matthews wrote:
> -- Forwarded message --
> From: Simon Sapin 
> Date: 30 September 2014 04:56
> Subject: Re: [dev-servo] Meeting notes 9/29 (CI; DOM binding safety;
> SpiderMonkey upgrade; test harness improvements; dogfooding; security;
> workweek; incremental reflow)
> To: Josh Matthews 
> 
> 
> https://github.com/servo/servo/wiki/Meeting-2014-09-29
> 
>> # Test harness improvements.
>> - zwarich: Every time I run tests and see hundreds of processes, I wonder
>> why we can't have one testrunner process that runs them in sequence. It's
>> what webkit and blink do.
>> - pcwalton: Yes, too much overhead right now.
>>
> 
> I happened to be talking with James about the test harness yesterday. The
> reason we start a new Servo process for every test is that it’s the only
> way to get Servo to navigate to another URL remotely. Other browsers use
> WebDriver for this.

And, for the record the reason that the wpt tests are driven remotely is
that they, unlike tests written specifically for servo, have to deal
gracefully with behaviour like crashes or hangs.

> We can probably fix this by implementing a subset of WebDriver. Reusing
> processes for running many tests requires some kind of wire protocol to
> communicate with them remotely, we might as well use a standard one.

Yeah, for running just js tests the requirements aren't too bad.
Basically you have to be able to use the protocol to load pages, execute
some script, and send some results from the script back to the remote
end. For reftests you also need to be able to take a screenshot. I guess
actually navigating browsing contexts in servo could be a problem?
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Error while testing

[James Graham]

On 06/10/14 11:14, Prasoon Shukla wrote:
> Hi all
> 
> I tried running the wpt tests (./mach test-wpt) and got the following error:
> 
> http://pastebin.com/f8kH8rah

This was a regression in the test runner that somehow went unnoticed.
Sorry about that. It should work now (you might want to delete the
_virtualenv directory to ensure you get the latest version).

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] HTTP referer disabled by default

[James Graham]

On 04/02/15 14:57, Manish Goregaokar wrote:
> I don't see any, however since Servo isn't being dogfooded yet, there's not
> much data we can gather on this (yet).
> 
> Still, it would be interesting to hide a bunch of "useful" web platform
> features behind some flags, add some specific reporting/logging, and see
> how many of them are really needed (and if they break important things)
> whilst dogfooding.

For general features you get much better data from adding telemetry to
an existing widely used engine like Gecko or Blink. Therefore the value
of doing this in Servo seems rather low. If there are features that we
think are rarely used and that we would prefer not to implement, running
experiments on Gecko seems like the easiest way forward (although I
think the details of the way Gecko's telemetry works makes it harder to
collect all the same data that is available through Blink infrastructure)

In this specific case the kind of question that needs to be answered is
more difficult since it's not of the form "is client side feature X used
at all". Unfortunately I guess that Servo is still a way away from being
able to help with that kind of question because there are so many
possible reasons for a site not working that it's hard to pin it on one
specific thing.
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

[dev-servo] Critic downtime

[James Graham]

Critic's server is undergoing a mandatory reboot for security patches at
on 2015-03-06 11:00:00 PM UTC. The anticipated downtime is less than two
hours, assuming everything comes back up as expected. During this period
critic will not be available for reviewing and will not pick up new PRs
or new commits from GitHub. Hopefully this doesn't cause any problems.
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Servo team Q2 projects

[James Graham]

On 18/03/15 17:32, Lars Bergstrom wrote:
> Here are some ideas we've been talking about on my team:
> 
> - Figure out CSS WG reftest integration. Right now, there's a
> python/mercurial build process that turns XHTML files into HTML to
> run them and it's not clear how to either integrate that without a
> giant mess (even by servo build standards!) or upstream work we do.
> I'd like to sort this out so we can have a test experience for layout
> tests that's similar to that of the DOM / WPT tests, especially for
> E-Easy bugs.

I think I have a concrete plan for this which isn't too complex. The
first step would be to put wpt (and CSS tests) in-tree directly in the
same way that Gecko does. Then the infrastructure for updating the
in-tree copy would be extended to also build the CSS tests. This might
be more messy than we would ideally like, but the complexity would be
contained to updates rather than imposed on all end users. Everything
could be run with wptrunner.

I can discuss whether this is something that I can work on, but if not
it's certainly something I can help with.

> - Critical features blocking many pages Need that flexbox. There were
> some scheduling snafus with it this quarter, but I think we need to
> be working on it in Q2 just to be able to build or view more pages,
> even if much of the team is working on perf/bugfixing. html5ever is
> going to need document.write for many pages to work.

I support working on document.write as early as possible; it adds a
surprising amount of complexity due to having non-trivial interactions
with the parser, script scheduling and document history.
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

[dev-servo] CSS tests now available (Was: Updating your repos)

[James Graham]

On 04/04/15 06:16, Manish Goregaokar wrote:
> git pull is going to be a tad broken for a bit due to the recent wpt changes
> 

And in case you were wondering *why* all this pain, that PR enables two
useful things:

1) CSSWG tests (primarily reftests) are now available through mach using
|mach test-css|. For now it's configured to just run the tests in the
CSS 2.1 testsuite, but we can add more in the future. These are not yet
turned on for bors, but that seems like a likely next step.

2) web-platform-tests moved in-tree. This means that it's now possible
to create patches in the servo tree to fix test issues and upstream them
without having to alnd them upstream and wait for the next sync. This
procedure unfortunately doesn't work for the CSS tests as they have a
build step.
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] PRs are now gated on csstests on linux

[James Graham]

On 02/06/15 18:46, Manish Goregaokar wrote:

Thanks to some great work by jgraham we've gated
 builds by the css tests (linux3).

The mac3 builder will continue to run css tests and will now become red
when the build fails. This will not block PRs though.

If you come across any intermittents let jgraham know (and disable them).
We can un-gate if there are too many of them.


Some additional details about running the tests:

In order to run the tests locally use |mach test-css|. Note that the 
tests take a while to run (particularly in debug) so for development you 
may be better off selecting a subset of the tests for the feature you 
are changing (related note: I think our test volume is now high enough 
that something like gecko's try server would be a big win).


If you need to update expectation data in order to get a PR landed the 
best approach is to use the raw log files from buildbot as input for the 
|mach css-update| command. These are visible on the waterfall page [1] 
as wpt_raw.log links, but note that you need to add /text to the url to 
get a plain text version. For example, one might update the metadata like:


curl 
http://build.servo.org/builders/linux2/builds/1914/steps/shell_1/logs/wpt_raw.log/text 
> linux.log
curl 
http://build.servo.org/builders/mac3/builds/76/steps/shell/logs/wpt_raw.log/text 
> osx.log

./mach css-update linux.log osx.log

This will make a local commit with the relevant metadata changes.

Thanks to larsberg and Manishearth for doing the ops side of this 
change, and generally being super-helpful.


[1] http://build.servo.org/waterfall

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] PRs are now gated on csstests on linux

[James Graham]

On 02/06/15 18:59, Jack Moffitt wrote:

The mac3 builder will continue to run css tests and will now become red
when the build fails. This will not block PRs though.


Are they unreliable there? What is the reason for this difference?


My new theory is that the problem on OSX is that the tests are running 
on two different slaves that are configured differently. I think if we 
fix this then we might get stable results.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

[dev-servo] Try server [was: Re: PRs are now gated on csstests on linux]

[James Graham]

On 02/06/15 19:04, Manish Goregaokar wrote:

As far as tryserver goes it should be easy to patch homu to accept a
wider list of "try-enabled" users, which we can hand out liberally like
Firefox does.

First we should get around to fixing the try feature though :)


Right, but the essential missing features compared to Firefox are that 
you don't have to open a PR and it doesn't compete for resources with 
integration builds (important if you want to e.g. test stability by 
rerunning test jobs N times).


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

[dev-servo] Reftest analyzer [was: Re: PRs are now gated on csstests on linux]

[James Graham]

On 02/06/15 18:59, James Graham wrote:


If you need to update expectation data in order to get a PR landed the
best approach is to use the raw log files from buildbot as input for the
|mach css-update| command. These are visible on the waterfall page [1]
as wpt_raw.log links, but note that you need to add /text to the url to
get a plain text version. For example, one might update the metadata like:

curl
http://build.servo.org/builders/linux2/builds/1914/steps/shell_1/logs/wpt_raw.log/text
 > linux.log
curl
http://build.servo.org/builders/mac3/builds/76/steps/shell/logs/wpt_raw.log/text
 > osx.log


Oh, and whilst I remember, I have a version of the gecko reftest 
analyzer, adapted for this log format at [1]. So you can upload the logs 
there and investigate the reason for the unexpected result by comparing 
screenshots down to the pixel level.


The tool will show all tests that fail, not just those where the result 
was unexpected; the latter are marked with a "!". This means it's also 
useful for investigating failures when working on layout code; just run 
the tests like


mach test-css --log-raw test.log 
tests/wpt/css-tests/css21_dev/html/some-feature-*.htm


and upload the resulting log file.

[1] http://hoppipolla.co.uk/410/reftest-analyser-structured.xhtml

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Multiprocess safety

[James Graham]

On 19/06/15 00:40, Patrick Walton wrote:


In an effort to keep this sort of thing from happening again, I'd like to
suggest that all new code that spawns threads and passes channels or boxed
objects over channels not be allowed to pass review until audited for
multiprocess safety up until the multiprocess branch lands.


As we discussed on irc, I'm happy with this, but would prefer that we 
get whatever's needed to replace the channels-over-channels pattern 
landed asap so we can get started replacing existing code before it 
actually breaks and don't end up in a situation where we're telling 
people that they can't land stuff because of multiprocess when 
multiprocess itself hasn't landed and we don't have an alternate design 
to offer.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Per-document event queues

[James Graham]

On 21/07/15 15:31, Josh Matthews wrote:


This appears to satisfy the requirements of the spec (c.f.
https://html.spec.whatwg.org/multipage/webappapis.html#processing-model-9 &
https://html.spec.whatwg.org/multipage/webappapis.html#task-source). My
main concern is the loss of deterministic ordering here - with a single
event queue, any event that is queued before another is guaranteed to
run before it as well. In this new model, if two events are added to
different queues in the same document, they could run in any order. This
could be fixed by always running the oldest event available in any
single document's queue, but are there dangers of out-of-order events
between same-origin documents? Thoughts?


Per the conversation we had on irc, I think that the spec allows the 
current model but not your proposed design (see [1]). I believe the 
proposed model has observable changes that likely have a compat. impact 
e.g. given a page with two subframes A and B, a script on the parent 
that does A.postMessage(1); A.postMessage(2); B.postMessage(3) can 
currently rely on the messages being received in the order 1,2,3 whereas 
that is not a possible ordering in the event-loop-per-document design. 
Therefore my initial reaction is that this is unlikely to work. However 
it might be worthwhile checking what IE does here as I vaugely remember 
they have relatively well-isolated iframes (but it's several years since 
I tested it, so I don't remember any details).


[1] https://html.spec.whatwg.org/multipage/webappapis.html#event-loop 
("There must be at least one browsing context event loop per user agent, 
and at most one per unit of related similar-origin browsing contexts.")

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] The rules of @bors-servo: retry

[James Graham]

On 03/08/15 23:41, Josh Matthews wrote:

Hello everyone! Let's talk about when and how I would like us to use
"@bors-servo: retry" in our PRs.

* Which tests actually failed? Is each one filed in servo/servo as a
known intermittent failure?
* For the tests which are not previously known intermittents, what is
their output on this run? Are there any errors or panics in the log that
look related to the PR's changes?
* Is there anything in the test itself that looks like the PR's changes
could affect it? Pay particular attention to tests that report an
unexpected pass.
* Have issues been filed for each test which intermittently failed for
reasons that cannot be ascribed to this PR's changes? Are these issues
in the form "Intermittent [timeout|fail|crash|pass] for
/path/to/test.html" with the I-intermittent label applied and the
complete output of that test from the failing run copied and pasted?
Build logs disappear; github comments are forever.
* Has justification been provided in this "@bors-servo: retry" comment
for why a retry is necessary. Preferably this happens in the form of "*
#[issue number 1], #[issue number 2]".


A lot of these issues look like "test results are hard to find". In the 
spirit of automation and tooling, it might be worth investing some time 
in making this easier. I can think of a couple of possibilities:


1) Write a bot that reads the structured log output and formats any 
unexpected results as a github comment (much like jdm-bot already does). 
For bonus point search for existing I-intermittents matching the test 
and result, and link that issue in the comment. For bonus bonus points 
instruct homu to do a single retry automatically if all unexpected 
results match known issues.


2) Start posting the results of test runs to treeherder, which will 
surface the failures in a clearer way than the buildbot waterfall and 
will hopefully in the near future get a measure of auto-matching test 
results against known intermittents (this will probably be internal 
rather than using any bugtracker as a backend, so it should be possible 
to also link specific intermittents to github issues if you want that 
feature).


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Reviewers

[James Graham]

On 01/09/15 12:29, Tetsuharu OHZEKI wrote:

Should we introduce some files which is like `OWNERS` file in Chromium
tree, or write a document to wiki?


If you want it in-tree — and that does have some advantages — it makes 
sense to make it easily machine readable using an existing format. Can 
you add arbitrary metadata to Cargo.toml files?


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] WPT Tests

[James Graham]

On 20/10/15 15:41, Gilles Leblanc wrote:

Hello,

Regarding the WPT tests which have a FAIL expectation, are these all
supposed to pass eventually or are some tests designed so that for them to
get the correct result, we need to have an expected failure ?


In theory all the tests are supposed to pass eventually (apart from one 
test Ms2ger added as a canary for bugs in the CI system). However it is 
of course possible that tests themselves contain bugs (often just 
because the spec changed since the test was written).


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[James Graham]

On 02/11/15 18:43, Josh Matthews wrote:

https://github.com/servo/servo/wiki/Meeting-2015-11-02


For the record, I'm very against trying to run mochitests in Servo. As I 
understand it the additional features it offers over wpt are mostly 
because it leverages gecko-internal APIs that Servo doesn't support. To 
the extent that it's necessary to support similar test-only APIs for 
Servo, I don't think trying to make them bug-compatible with the Gecko 
implementations is a worthwhile goal; better just to implement the 
features that we need WebDriver or as servo-only DOM APIs and writing 
web-platform-tests, even if we can't share all of them. In the case we 
can share them, of course, that multiplies the value of the test 
enormously since we can also run it in Gecko, Blink, Edge, etc.


The js API for writing interactive tests through WebDriver is a 
contributor project right now, and we also have preliminary interest 
from Google in collaborating on this effort so they can also write tests 
in this style, so I expect to see something usable in the near future.


In terms of Servo work, this will require more WebDriver support to be 
implemented; I think a reasonable fraction of what's missing would be 
Good First Bug material.

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[James Graham]

On 03/11/15 22:08, Robert O'Callahan wrote:


Why not create a Mochitest compatibility layer over testharness.js so
that tests that only use SimpleTest.waitForExplicitFinish(),
SimpleTest.finish(), is(), todo() and ok() can run on Servo with trivial
changes?


A few reasons I am skeptical of this:

* Mochitests test gecko behaviour, not standardised behaviour. These can 
and do differ. Therefore they are even less trustworthy than the average 
test.


* Mochitests are really written assuming the full gecko featureset. 
Given the total lack of isolation between asserts, they may behave 
rather unexpectedly and misleadingly in a browser not having all those 
features.


* In the long term having multiple APIs for writing tests that people 
have to learn in order to read tests is a big net negative. If we allow 
mochitests to be upstreamed with a shim we should expect other vendors 
to do the same, and to end up with half a dozen possible test formats.


In addition, I don't think that using mochitests which don't use 
internal APIs helps at all with the original problem of testing things 
like dynamic style changes that require internal-only APIs to trigger.

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[James Graham]

On 04/11/15 04:52, Manish Goregaokar wrote:

* In the long term having multiple APIs for writing tests that people have
to learn in order to read tests is a big net negative. If we allow
mochitests to be upstreamed with a shim we should expect other vendors to
do the same, and to end up with half a dozen possible test formats.



What if we translate instead of shimming? At least upstream all
spec-conformant browser-agnostic tests.


Well certainly, taking a mochitest, removing any gecko/spidermonkey 
specific assumptions or non-standard features, and rewriting it as a 
web-platform-test seems unarguably good. It also seems like it will be a 
non-trivial amount of work. Worth doing, but not exactly a source of 
free tests.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[James Graham]

On 04/11/15 10:24, Robert O'Callahan wrote:

On Wed, Nov 4, 2015 at 5:14 PM, James Graham  wrote:


On 03/11/15 22:08, Robert O'Callahan wrote:

Why not create a Mochitest compatibility layer over testharness.js so

that tests that only use SimpleTest.waitForExplicitFinish(),
SimpleTest.finish(), is(), todo() and ok() can run on Servo with trivial
changes?



A few reasons I am skeptical of this:

* Mochitests test gecko behaviour, not standardised behaviour. These can
and do differ. Therefore they are even less trustworthy than the average
test.



I'm not sure what you mean by this. It's easy to write web-platform-tests
that depend on non-standard behavior. I had to debug one such last week.
Furthermore Gecko mochitests have been tested against Gecko, which has
itself been tested on the Web, so Gecko mochitests are far more likely to
be "correct" than a test someone just wrote and only tested on, say, Servo.


I mean that mochitests are used as regression tests; they aren't 
necessarily testing the per-spec behaviour (which might match Blink or 
Edge, but not Gecko), but whatever the Gecko code implements. Of course 
web-platform-tests can be buggy like any other code, but they are at 
least intended to test the right thing.



* Mochitests are really written assuming the full gecko featureset. Given

the total lack of isolation between asserts, they may behave rather
unexpectedly and misleadingly in a browser not having all those features.



Many use Gecko-specific features but many don't.


Not just use of gecko-specific (or, often spidermonkey-specific) 
behaviours, but also the fact that mochitests assume an assert never 
fails (because if one does it's a regression that needs to be fixed). In 
Servo, which might not implement all the features used, an assert can be 
expected to fail for some time until more feature work is done. This is 
particularly true at the moment when Servo doesn't implement the window 
error handler. I anticipate this impedance mismatch would cause the 
tests to be less useful in Servo than they are in Gecko.



* In the long term having multiple APIs for writing tests that people have

to learn in order to read tests is a big net negative. If we allow
mochitests to be upstreamed with a shim we should expect other vendors to
do the same, and to end up with half a dozen possible test formats.



That's fair, though it's a problem we introduced when we started adding
web-platform-tests to the tree.

Put it this way: when Servo needs tests for a feature, I expect it makes
much more sense to take Gecko mochitests for the feature and convert them
to web-platform-tests than to write new tests from scratch.


Well sure, I agree that taking mochitests as the input to a test-writing 
effort is a good idea. I see this as being very different to blindly 
shimming mochitests into the wpt harness. Having said that, however, I 
don't think people have complained a lot about lack of test coverage 
from wpt except in the areas that it doesn't cover at all i.e. dynamic 
changes to layout, or other human interaction.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[James Graham]

On 04/11/15 11:12, Robert O'Callahan wrote:

Well sure, I agree that taking mochitests as the input to a test-writing
effort is a good idea. I see this as being very different to blindly
shimming mochitests into the wpt harness. Having said that, however, I
don't think people have complained a lot about lack of test coverage from
wpt except in the areas that it doesn't cover at all i.e. dynamic changes
to layout, or other human interaction.



To pick a couple of areas I work on: for CSSOM-Views for example there is
practically no coverage at all. The media tests are better but still very
limited compared to what we test in mochitests.


It would be great to have a record of some areas where we know that 
web-platform-tests has missing coverage; we occasionally get people 
looking for areas where they can make useful test contributions and 
asking what's required. Can you file some issues on GitHub, possibly 
pointing to the relevant mochitests that we could draw from?


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes 11/2 (review carry-over; test coverage; 2016 roadmap; rebase/autosquash; PR queue; debug logging; CSSWG reftests)

[James Graham]

On 04/11/15 11:41, Robert O'Callahan wrote:


Sure. https://github.com/w3c/web-platform-tests/issues/2304


Thanks!


The media tests are better than I thought --- I found more. They don't test
the variety of problematic media files that our mochitests do, but maybe
that's not in scope? Should we be worried that regressions in the media
stack seldom trigger W test failures on mozilla-inbound but often trigger
mochitest failures?


I consider anything that might affect browser interop or quality 
in-scope, as long as it can be run cross-browser and it's clear what the 
expected result is.

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] IndexDB project

[James Graham]

On 15/12/15 03:20, Shing Lyu wrote:

Hi,

I'm Shing from the Mozilla Taipei office. I asked Lars for interesting but
not urgent (don't want to block critical feature) during the Mozlando
workweek, he suggest me to look into IndexDB support.

I wonder if anyone can help me with
* Split it into smaller work items
* Point me to previous patches

I also know some folks in the Taipei office are interested in contributing
to Servo. So I think we might be able to form a small project group so we
can help each other.

Any suggestions or comments?


I don't suppose the uncertainty in the rest of the thread left you 
thinking "maybe I should implement document.write instead", did it? :)


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Using Cargo in Gecko

[James Graham]

On 17/12/15 23:04, Valentin Gosu wrote:


 From what I understand servo's mach has a command for two-way sync between
web-platform-tests and upstream.


It does, but let's be clear that the implementation there isn't end-user 
quality and is simplified byd specific policies in web-platform-tests 
that allow us to automatically merge PRs when syncing upstream. So that 
particular tool isn't an existence proof that two-way syncing is a 
solvable problem in general.


I'm not saying that it isn't, of course. But I would at least expect 
such a tool to be quite complex to get right, and would be wary of 
holding back on anything until it's done.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Using Cargo in Gecko

[James Graham]

On 21/12/15 22:45, Yehuda Katz wrote:

Yehuda Katz
(ph) 718.877.1325

On Mon, Dec 21, 2015 at 2:28 PM, Josh Matthews 
wrote:


On 2015-12-21 5:21 PM, Yehuda Katz wrote:


On Mon, Dec 21, 2015 at 2:14 PM, Bobby Holley 
wrote:

I don't think this is going to fly in Gecko, unfortunately.


Gecko is a monolithic repo, and everything needs to be vendored in-tree
(a
non-negotiable requirement from the build peers). This means that we'll
already need an automated process for pulling in updates to the shared
code, committing them, and running them through CI.



Can you say a bit more about what the requirements are here? Is the reason
for including the code in-tree that you need to be 100% confident that
everyone is talking about the same code? Or is it more than that?



The "non-negotiable requirement from the build peers" is that the machines
that build Firefox do not have internet access.



That's a totally reasonable requirement and one that I am very comfortable
saying that Cargo should support.

To support this requirement, we plan to add a new command that packages up
an application into a single tarball (or equivalent), complete with all
dependencies, and another command that can build such a package without
network connectivity.


It is not immediately apparent to me if that is sufficient to meet the 
Gecko requirements, but again, having this conversation on a list 
without gps, ted, glandium and other build peers seems rather 
counterproductive since they know both the requirements that we have 
from our current build system, and the improvements that are in the 
pipeline for next year (which I understand to be numerous and considered 
a priority). Without this information it's impossible to tell if any 
proposed change to cargo is solving the right problem.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Using Cargo in Gecko

[James Graham]

On 22/12/15 03:52, Yehuda Katz wrote:


In my opinion, #3 is best addressed with support for automated mirroring of
both crates.io and git dependencies. Mega-corps like LinkedIn use mirroring
services for Rubygems, npm, and maven to ensure high-availability. This
maintains the normal developer workflow and ensures that developers on the
project do not rely on custom infrastructure for updates, while still
ensuring that the deployment process isn't dependent on third-party
infrastructure.


It's not terribly automated, but we have an internal PyPI mirror for 
various things that are used in the build process. My experience is that 
every time you have to use this it's enough pain, both in the short term 
and for people doing maintenance after you, that you should just put the 
dependency directly in-tree instead.

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Proposed work for upcoming sharing of Servo components with Firefox

[James Graham]

On 21/06/16 13:34, Lars Bergstrom wrote:

On Jun 20, 2016, at 9:25 PM, Boris Zbarsky  wrote:


On 6/20/16 8:35 PM, Lars Bergstrom wrote:

Backouts came up in the discussion, and I tried to make it pretty
clear that they are not compatible with a GitHub-style
development model (even if you *could* hypothetically do crazy
things like force-pushing rewritten histories to master).


Wait, why are rewritten histories relevant?  A backout doesn't
involve rewriting history...


If a backout lands as a `git revert` of the offending commit(s) I'm
certainly less concerned, as that's annoying but not impossible for
people to continue rebasing against.

The way it was explained to me (and I may have been
misunderstanding!) is that if a batch of changes land on m-i but then
fail tests, they are removed and there is the mercurial equivalent of
a force-push back to m-i without those changes in the history. That
workflow made me concerned for people with an open PR, as if the PR
was based on master at commit A, commit B lands and they rebase
against it, but then B disappears from the history, getting their PR
landable again is likely going to require intervention by a git
expert.


There has been a communication breakdown. What happens on m-i is the 
equivalent of |git revert|.


In general my main concern for people contributing through the GH UI 
would be whether a reopening the PRs would work. I guess a bot could 
make it happen (but rebasing on top of a revert is a little tricky 
because git tends to assume the patch already landed, so you need to 
cherry-pick instead).

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Proposed work for upcoming sharing of Servo components with Firefox

[James Graham]

On 22/06/16 14:27, Boris Zbarsky wrote:

1)  One side (the wpt side) has pretty trivial integration tests: it
basically just tests filename length.  So if Gecko-side changes break
those tests fixing is not too terrible.

2)  The other side (the gecko side) has a way to annotate failing
things, and whatever starts failing after a change on the wpt side is
just annotated as failing and checked in that way.  I think this is
pretty broken, but that's what we're doing right now...

Needless to say, something like the style system would have somewhat
different constraints on whether it actually needs to pass tests (for
real, not for purposes of the test harness; just annotating stuff as
"known fail" and moving on is not really what I anyone is looking for
here).  But even with the behavior above wpt merges are a huge
time-sink.  Just talk to jgraham.


Yeah. I'm not really sure how we would do better than annotating 
expected results with wpt merges, although we can for-sure do better at 
making the failures we are importing more visible. But the way the sync 
works is clearly not optimal; downstreaming in batches is probably 
necessary to allow the expectation data to be merged, but we should 
switch to a system where upstreaming happens when the patch lands on 
m-i, to reduce the chance of merge conflicts (at the moment I have to 
solve these manually, and fixing conflicts in code you have never seen 
before is not ideal). Of course making this change would lead to the 
possibility of backouts, which probably shouldn't be reflected upstream.


Anyway, that was somewhat off-topic, but insofar as I had a point 
related to the original discussion it was "merge little and often, and 
do everything you can to avoid merge conflicts, because they will make 
you sad".

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Questions about constellation, sandboxing and multiprocess

[James Graham]

On 04/08/16 12:24, Olaf Buddenhagen wrote:

Hi,

On Wed, Aug 03, 2016 at 06:22:51PM +0200, Till Schneidereit wrote:


I'm not concerned about code complexity, but about memory usage. Memory
usage in many-tab scenarios is one of the measures where Firefox is still
vastly superior to the competition, and I think we should aim for roughly
matching that.


I'm surprised that I keep hearing this claim. It doesn't at all match my
real-world experience from not too long ago doing my everyday browsing
on a serverely memory-constrained system...


Fortunately we have data! [1]. To quote from the conclusion there 
"Simply put: the more content processes we use, the more memory we use". 
So I don't know why your experience doesn't match, but both in theory 
and when measured memory usage should increase with the number of 
content processes.


[1] 
http://www.erahm.org/2016/02/11/memory-usage-of-firefox-with-e10s-enabled/


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] The size of the tests/ directory

[James Graham]

On 25/08/16 08:56, Ms2ger wrote:

On 24/08/16 18:12, Lars Bergstrom wrote:

Currently, the GitHub Servo repository has 134,062 files... 131,477 of
which are in the tests/ directory. Of those, 102,871 are in the CSS WG
tests. I know it's not a perfect measure, but that's also 862megs of
the 1.132gb of disk space usage reported by `du` on macOS for a Servo
checkout.

So, I'd like to have a discussion to see if there's something we can
do here to reduce the number of files. There are two reasons this has
come up:
- If we do end up "copying" the Servo GitHub repository into the
Mozilla monorepo, unfortunately mercurial cannot scale to that number
of files (though there is work going on at Facebook to make it able to
do so in the next year or two). Plus, the non-CSS WG tests in the WPT
directory are duplicate with those already in m-c.
- That's a lot of disk space, and might not be needed for new
contributors who working on anything other than layout features.

There are a few options I can see here (apart from "do nothing"):
1) Reduce the duplication in the CSS WG tests. A lot of it is due to
the build system. How small could we get it? Similar to WPT (i.e., 10x
fewer files)?
2) Break out the CSS WG and WPT tests into a separate location, to be
downloaded as-needed. If we do this, though, what is the workflow for
adding, deleting, or correcting a test? And is it still just as easy
to run a single test? Jamming it in a crate will leave it in a wonky
directory (e.g., target/debug/build/csswg-7062634bbf237306/output)
that is not exactly awesome for viewing & running tests from.

Any opinions or other ideas here?


I vehemently object to moving wpt out of tree; that would significantly
discourage people from writing tests, especially tests that can be
shared with other browsers. (Shared tests are the single best way of
ensuring we are compatible with other browser engines, and they with us.)

The csswg tests have a number of issues resulting from the way their
build system works, but there's little point in spending resources
there; the built copy of the tests will be removed in the coming months,
as the unbuilt version will be added to wpt. (At that point: see the
paragraph above.)


I agree with Ms2ger. Maybe I should just make that my sig :) I think 
it's easy to focus on the cost of these files and neglect the immense 
value that they bring. AIUI once the build system is gone it will be 
possible to directly contribute css tests upstream like we do with any 
other kind of wpt tests.


As Simon estimates, without the build step, the number of files is 
reduced by a factor of ~3. If that still isn't good enough it is 
possible to consider hacks like never importing tests for specifications 
that servo doesn't support yet, but those are intrinsically short-term 
gains. There may also be some smaller, high-effort, but longer term 
gains from things like merging (layoutwise) duplicate reference files. 
However if servo/servo doesn't end up in m-c that seems like a 
lower-value proposition (especially given that all of these files will 
be in m-c anyway).



___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Servo testing as part of PhD dissertation

[James Graham]

[not sure if this will make it through to the list]

On 06/09/16 21:35, Jack Moffitt wrote:

I haven't quite settled on my dissertation topic, but my top contender at
the moment involves property-based (i.e. QuickCheck style) generation of
random web pages/stylesheets.


A sort of subtask of this which would be extremely useful is taking a
known rendering problem and producing a minimal reproduction of it.
For example, many issues are discovered in existing pages with perhaps
hundreds of kilobytes of extraneous data. It would be nice to reduce
the failing example to a minimal size. One issue is how to make an
oracle here. It would probably be an improvement to have it be only
semi-automated, where it does some shrinking then asks and repeats.


There is some prior art here e.g. [1]. I wrote a similar tool that was 
specialised to reducing js code whilst at Opera, but that doesn't ever 
seem to have been released. In both cases you either had to write a 
one-off function to determine if the testcase was a pass or fail, or 
have a human judge it. Obviously the latter is impractically slow if 
your input is large.



The oracle would be a cluster of browsers
(multiple vendors/variants) driven by WebDriver/Selenium that would render
the test cases and screenshot them. Significant discrepancies between
renderings would be considered a failing test case and then standard
QuickCheck-style shrinking would be used to reduce the test case HTML/CSS
to a minimal-ish reproducer.


Each browser renders things slightly differently, so pixel by pixel
comparison across browsers is probably not going to work well. For our
own testing of this kind we instead produce the same result using two
different techniques, or in a few cases we make reference images.
However making reference images can't account for all rendering
differences (like text) and so we avoid it if possible. I imagine it
would be quite difficult if the reference image was from another
engine, not our own.


Yes, I imagine specifically font rendering will be a problem, along with 
antialiasing in general and tegitimate-per-CSS variations in properties 
such as outline.


However I think you might make progress with some sort of 
consensus-based approach e.g. take a testcase and render it in 
gecko/blink/webkit/edge. If the difference by some metric (e.g. number 
of differing pixels, although more sophisticated approaches are 
possible) is within some threshold then check whether servo is within 
the same threshold. If it is consider that a pass otherwise a fail.



Is this idea of interest to the Servo team? Would it be useful for Servo
development/testing? Or perhaps redundant with existing testing I'm not
aware of?


The main kind of testing we do is reference testing where the
reference is the same content achieved by different means. This is
pretty robust to things like font rendering changing slightly between
versions. We have some JS level testing where JS APIs are invoked and
then results verified, but it sounds like you are more focused on the
visual testing aspect. As an aside, I think quickchecking JS  APIs is
likely to find a ton of bugs and be useful too, plus it probably
doesn't have the oracle problems.


But this is also a good idea :)

[1] 
http://www.squarefree.com/2007/09/15/introducing-lithium-a-testcase-reduction-tool/

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Proposal: move the source code for rust-selectors into servo/servo

[James Graham]

On 09/02/17 09:51, Anthony Ramine wrote:


I haven't split it out yet, because I'm still hoping that the
outcome of this discussion is that I won't have to. The unified
code changes are in part 3 of
https://bugzilla.mozilla.org/show_bug.cgi?id=1336646


How does the discussion mean you don't have to split? All I see is
three patches, none of them making it obvious how the selectors API
changed. This also means that the commit messages will never ever be
written from the POV of selectors, and always from the POV of servo,
thus quite limiting any lingering hope to have external contributors
now, given it will not be as discoverable as before what is happening
in the crate at the commit level.

In general, I posit that Mozilla completely, utterly, lost against V8
in the embedded department in a huge part because of this
mono-repository thing where everything is muddled together.

We can't quantify the missed opportunities when merging things
together in a single repository.


I don't really have any skin in this game, but I will note that the 
web-platform-tests have been much more successful since we allowed 
people to use them in the same repository as their browser code, even 
though it does mean we get a lot of commits with useless commit messages 
like "tests". So it's not always the case that the unquantifiable 
benefits of multiple repositories always outweigh the unquantifiable 
benefits of a single (from the point of view of a specific group of 
developers working on a larger project) repository.

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Proposal: move the source code for rust-selectors into servo/servo

[James Graham]

On 09/02/17 16:18, Michael Howell wrote:

WPT still has its own repo; the Servo repo just does an occasional
bidirectional sync. The equivalent for rust-selectors would be if the
rust-selectors repo was kept and just occasionally synced with Servo (like
Servo is doing for M-C, anyway).


Sure, but my point was that making the workflow for regular contributors 
as painless as possible (by allowing them to work in a "monorepo" setup) 
vastly increased the number of contributions we got and was a huge win 
for the project. Of course the specifics of wpt demand a complex 2 way 
sync architecture to support that, since there are multiple downstream 
projects using the tests.


___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

[dev-servo] wpt tests enabled for linux64-stylo

[James Graham]

I jut pushed bug 1370461 to autoland, which is expected to enable wpt 
tests for linux64-stylo builds. These jobs will be Tier-2 like other 
stylo test jobs. The associated try push is [1]


If you need to set stylo-specific test expectations, there is a boolean 
variable 'stylo' which can be set in the expectation metadata files in 
the normal way for a web-platform test.


[1] 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=94deac4036000ceb59a1582d526577f5cbbdb573

___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Categorizing Stylo reftest failures

[James Graham]

On 10/08/17 02:53, Manish Goregaokar wrote:

https://gist.github.com/Manishearth/086118c940ff86a6cfc573f53c508279



FWIW I made a list of the differences between reftest results in 
web-platform-tests with stylo enabled vs not (based on the expectation 
metadata files). Paths are relative to the root of web-platform-tests.


https://pastebin.mozilla.org/9029321
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Enabling tests in web-platform-tests/css/

[James Graham]

On 01/11/17 21:25, Josh Matthews wrote:
In short, I think we should enable the new testsuite, remove the old 
tests and related CI jobs, and never look back! The ability to upstream 
reftests and share others' improvements will be a significant boon.


Hurrah! Thanks for doing this.
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo