Re: Proposed policy change: reusability of tests by other browsers
On Sun, Nov 11, 2012 at 4:21 PM, Neil n...@parkwaycc.co.uk wrote: Why not simply verify that the list of actual fails equals the list of expected fails, and report items that are only in one of the two lists? That would be a bit more robust, yes, and it should be doable without much work. It still wouldn't detect the case where different sets of passing tests are run, such as because a change accidentally disabled a large chunk of tests. The system James seems to be describing would detect that condition too. On the other hand, all our existing mochitests don't detect it and we seem to do fine. (There *are* cases where someone accidentally disabled a bunch of tests and no one noticed. Some months ago I found a number of those when I changed mochitests to fail if the file ran no tests. But that only detects if all tests in the file are disabled, not if only a handful are disabled. Still, it's not a common enough scenario that it's worth spending much effort on, IMO.) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Sat, Nov 10, 2012 at 9:41 AM, Boris Zbarsky bzbar...@mit.edu wrote: I believe right now we have a list of known failures alongside such tests, and our own test harness knows to compare what the tests are reporting to our list of known failures. As in, we're not using the pass/fail state of the tests directly; we're comparing it to should all pass, except this whitelist of things that we know fail. Constructing these whitelists of known failures is indeed a bit of a PITA, but they're pretty static until we fix stuff, usually. Yes, exactly. And they're quite easy to construct -- there's a script these days (parseFailures.py) that you run on the output of the mochitests, and it creates all the directories and files for you. The code for our testharness.js wrapping is here: http://hg.mozilla.org/mozilla-central/file/ea5c4c1b0edf/dom/imptests See the README. To import a new test suite, all you have to do is add a line to a file (or a new file) specifying the location of the test suite and the directories that are wanted, run the importTestsuite.py script, run the test suite to get a list of known failures, and use parseFailures.py on the result to generate appropriate JSON files in the failures directory. It's only a minor hassle. Currently we only check that no test fails that's not on the per-file whitelist of expected fails, and in practice that works fine for us. If we wanted to be pickier, we could list all expected results, both pass and fail, and verify that the lists exactly match. This is unpleasant in practice because some of the test files I wrote run tens of thousands of tests, which leads to JSON files quite a few megabytes in size that have to be loaded to run the tests. Since in most files we pass all or almost all tests, storing only failures is a very effective optimization. It's certainly true that if a file threw an exception at file scope, it would make the test useless. However, if we can't change the file, there could be all kinds of things about it that are broken anyway. For instance, it could wrap many unrelated things in a single test(), and then you have exactly the same problems. For the time being, any tests we import are tests we can change -- they're all from the W3C and most are written by Mozilla contributors. So if it doesn't play nicely for our system, we can always change it. I expect that to continue to be the case. If an odd file breaks and we can't change it for whatever reason, we can live without the test coverage. It's not a problem we have in practice. So I still don't see the value in the test/assert model used by testharness.js, as opposed to having everything be one big test but asserts be non-fatal. In particular, what value does test() provide that couldn't be provided about as well by try/catch blocks, aside from aesthetic preferences about grouping? ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 11/9/12 12:52 AM, James Graham wrote: I know Mozilla use a system where all the tests in a file should pass, but I don't see how that will work well when you don't control the tests. If you are manually editing every file when you import it, I imagine that updating tests will be so time consuming that it will be tempting not to bother. How do you plan to address this? I believe right now we have a list of known failures alongside such tests, and our own test harness knows to compare what the tests are reporting to our list of known failures. As in, we're not using the pass/fail state of the tests directly; we're comparing it to should all pass, except this whitelist of things that we know fail. Constructing these whitelists of known failures is indeed a bit of a PITA, but they're pretty static until we fix stuff, usually. -Boris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 11/07/2012 02:03 PM, Henri Sivonen wrote: On Wed, Oct 10, 2012 at 10:46 AM, Aryeh Gregor a...@aryeh.name wrote: That said, of course, Mozilla hackers *are* familiar with Mochitest but not testharness.js, and adopting testharness.js in parallel with Mochitest would require people to be familiar with both. That is certainly a minus. I was told at TPAC that testharness.js has gotten/is getting a mode where you get to make the entire file one test from the point of view of testharness.js. In that case, the user experience of testharness.js is (I'm told) isomorphic to using mochitest with explicit finish. There is an experimental branch with this mode in; it isn't production quality yet. I am still unsure that it's a good idea; in particular I think it encourages people to write multiple tests on the same page in such a way that if one fails the whole suite stops running. This seems very unpleasant to deal with; remember that unlike typical Mozilla tests these will not necessarily all pass in their unmodified form, and that modifying imported tests inline creates a headache when you want to merge changes. Nevertheless I will put the current code somewhere public and let people experiment with the API. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Wed, Oct 10, 2012 at 12:46 AM, Jonas Sicking jo...@sicking.cc wrote: However, for someone working against a looming deadline, the cost of writing sharable tests and risk missing the deadline can be much higher than having those tests not be run by other browser vendors. Which is why we have rolling releases, right, so things can be pushed off a bit if necessary to get them right? If individual projects' managers set deadlines that don't prioritize portable testing, then of course those projects will have trouble getting portable tests done, but that's true for any possible goal. Many software projects don't require good testing or QA at all, and as a result developers write half-baked features so they can meet deadlines that don't allow enough time to do things right. That doesn't mean we should avoid doing things right! Naturally, we do have to prioritize our resources, and sometimes it will make more sense to not bother with reusable tests if they're too much work. That doesn't mean we shouldn't aim to *ever* write reusable tests. Additionally, I think that the more overhead the harness has, the less thorough the tests will be. I've definitely noticed that the tests from Opera for CORS has been a lot less thorough than the tests that I wrote myself. I absolutely think that there's a stronger bias against writing more comprehensive tests the more work is needed to write those tests. No amount of rules forcing developers to write tests against testharness.js will remove that bias. I wrote a lot of tests using testharness.js that I hope you'll agree are pretty thorough, such as Range and Selection tests: http://hg.mozilla.org/mozilla-central/file/5cca0408a73f/dom/imptests/webapps/DOMCore/tests/approved http://hg.mozilla.org/mozilla-central/file/5cca0408a73f/dom/imptests/editing/selecttest On the other hand, I've seen *manual* tests (= no framework overhead at all) submitted to the W3C by Microsoft that are so superficial as to be worse than useless, not to mention some that were wrong. That you write better tests than other people can't be blamed on the framework! In practice: I've written many tests, both complicated and simple, in both Mochitest and testharness.js. Once you get used to them, they're about equally easy to write, although testharness.js winds up being slightly longer. If you're writing any sort of nontrivial test and you're familiar with the test harness you're using, 90% of your effort is going to be thinking about what exactly to test and how to test it, not actually typing stuff. That said, of course, Mozilla hackers *are* familiar with Mochitest but not testharness.js, and adopting testharness.js in parallel with Mochitest would require people to be familiar with both. That is certainly a minus. So I'd much rather spend the additional effort to create a test harness which is optimized for getting comprehensive tests than take the additional cost to developers and the reduction in test quality that comes with a overly heavyweight harness. I think making Yet Another Test Harness is not the right way to go. testharness.js has traction in the standards world, it's well documented, it already works in all browsers, at least Mozilla and Opera (possibly MS/WebKit too) can already use testharness.js tests internally, and there are people from every major browser who are at least somewhat familiar with it. It may not be perfect, but it doesn't make sense to reinvent the wheel still again unless there are really compelling specific flaws that can be identified. This doesn't match my experience at all. My experience is that writing tests has a high cost and results in fairly complex test files. It depends. If you're writing tests for an entire feature, then yes, it's very complicated. I've done that -- Range, Selection, and editing, for instance. But most routine bugs are simple things and the tests are just a few lines long. Even if we only shared those tests, it would be a step forward. On Wed, Oct 10, 2012 at 6:41 AM, Boris Zbarsky bzbar...@mit.edu wrote: On 10/10/12 12:23 AM, Ian Bicking wrote: Here's how I think you'd write a simple XHR test in both: // SimpleTest aka MochiTest req = new XMLHttpRequest(); req.open(GET, /example.json); How did example.json get there? What if you need to test CORS? With mochitest at this point you're doing some SJS work and whatnot. // testharness And here you have to go and do whatever is appropriate to your server setup. Which is not part of testharness. Which is why the CORS tests imported from Opera to the W3C ended up all broken, because they did not configure the server correctly. testharness.js absolutely does not support all the same features as Mochitest. It also doesn't support taking screenshots, for instance, which makes it useless for certain things. But we can still use it for the things it does support, like self-contained DOM tests. Although yes, this means using a mix of
Re: Proposed policy change: reusability of tests by other browsers
On Wednesday, 10 October 2012 07:18:45 UTC+2, Boris Zbarsky wrote: On 10/10/12 1:13 AM, Ian Bicking wrote: OK – so if I understand the objection to testharness isn't anything in testharness.js itself, but that it's an incomplete solution as it doesn't define an environment? That's _my_ primary objection, after looking at it briefly and seeing how it works in practice. Do you have a concrete suggestion for how to improve this, in a way that works cross-browser? AIUI (which is not very well, so please correct me if I am wrong), the Mozilla solution involves running a custom XPCOM-based web server, and thus is not very portable. The W3C solution is basically there exists a server with a set of known subdomains and the ability to run PHP. When I have wanted to write tests for things that require multiple origins, I have assumed that the domain from which the tests are being served is unknown, but that there are sure to be some specific subdomains. This is far from ideal, but does allow me to make tests that are portable across our infrastructure and the W3C infrastructure. It also seems like they could, in principle, be ported to other infrastructure. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Sat, Oct 6, 2012 at 6:25 AM, Jonas Sicking jo...@sicking.cc wrote: In general, testharness.js seems to be more optimized for producing a result report which measure how close an implementation is to implementing a feature, than it is optimized for making it easy to write tests. I think it's actually optimized for Opera's internal testing needs as much as anything, since James Graham is the one who wrote it. Opera has a much more elaborate testing infrastructure than we do. I think they actually have stuff like databases instead of grepping plaintext output, and the ability to run individual test files, and automatically running new test files X times to record all intermittent failures. If we just layer it on top of mochitest, we can't take advantage of the greater structure, so the added complexity is wasted. James has indicated that features like each file runs the same set of tests no matter how many fail are essential to working properly with Opera's test setup. I believe many developers right now end up spending as much time writing tests as they do implementing features. That is a very high cost, but something that is definitely worth it. However we should be working towards lowering that cost rather than increasing it. Rather than trying to convince developers that testharness.js would in fact not increase the cost of writing tests, I think we should try to get W3C to adjust testharness.js such that it's easier to write tests for it. If we make it as easy to write W3C tests as it is to write mochitests then I would absolutely agree with your proposal. I would imagine that would also make it easier to get other browser vendors to do the same, as well as members of the webdevelopment community. I think testharness.js would definitely increase the costs of writing tests. I think the increase in interop would be worth it, if other browsers start using our tests. It's all good and well if we implement a feature correctly, but if other browsers implement it differently, it's not very useful to authors. This is why we want to submit tests to the W3C to start with. Another problem that I think we'd have is that many of our tests use generators and yield. This *dramatically* cuts down on the complexity of writing complex tests which has lots of asynchronous callbacks. For example [1][2] would have been much harder to write without them. I think our approach here could be migrate these tests to use ES6 based generators as soon as we have them implemented in gecko, and then submit them to W3C as soon as enough browsers implement ES6. I don't think that we should be telling people to not use generators in the meantime. My experience is that rewriting tests to use generators both cuts down on the test writing time, and makes it much less likely that the test ends up with intermittent orange bugs. If it's a pain to write a particular file in testharness.js, it can be kept as mochitest. In my experience, quite a lot of tests boil down to like ten lines, which would take about three minutes more to write using testharness.js than mochitest. Also, a test that's based on testharness.js but uses some Gecko-only features would be easier to make portable later than a test that's based on mochitest and also uses Gecko-only features. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Tue, Oct 9, 2012 at 2:43 AM, Aryeh Gregor a...@aryeh.name wrote: On Sat, Oct 6, 2012 at 6:25 AM, Jonas Sicking jo...@sicking.cc wrote: In general, testharness.js seems to be more optimized for producing a result report which measure how close an implementation is to implementing a feature, than it is optimized for making it easy to write tests. I think it's actually optimized for Opera's internal testing needs as much as anything, since James Graham is the one who wrote it. Opera has a much more elaborate testing infrastructure than we do. I think they actually have stuff like databases instead of grepping plaintext output, and the ability to run individual test files, and automatically running new test files X times to record all intermittent failures. If we just layer it on top of mochitest, we can't take advantage of the greater structure, so the added complexity is wasted. James has indicated that features like each file runs the same set of tests no matter how many fail are essential to working properly with Opera's test setup. I believe many developers right now end up spending as much time writing tests as they do implementing features. That is a very high cost, but something that is definitely worth it. However we should be working towards lowering that cost rather than increasing it. Rather than trying to convince developers that testharness.js would in fact not increase the cost of writing tests, I think we should try to get W3C to adjust testharness.js such that it's easier to write tests for it. If we make it as easy to write W3C tests as it is to write mochitests then I would absolutely agree with your proposal. I would imagine that would also make it easier to get other browser vendors to do the same, as well as members of the webdevelopment community. I think testharness.js would definitely increase the costs of writing tests. I think the increase in interop would be worth it, if other browsers start using our tests. It's all good and well if we implement a feature correctly, but if other browsers implement it differently, it's not very useful to authors. This is why we want to submit tests to the W3C to start with. I agree that at a large scale, the additional value from writing tests that are sharable with other browser vendors and with the W3C community is technically worth the overhead that testharness.js brings. However, for someone working against a looming deadline, the cost of writing sharable tests and risk missing the deadline can be much higher than having those tests not be run by other browser vendors. Additionally, I think that the more overhead the harness has, the less thorough the tests will be. I've definitely noticed that the tests from Opera for CORS has been a lot less thorough than the tests that I wrote myself. I absolutely think that there's a stronger bias against writing more comprehensive tests the more work is needed to write those tests. No amount of rules forcing developers to write tests against testharness.js will remove that bias. So I'd much rather spend the additional effort to create a test harness which is optimized for getting comprehensive tests than take the additional cost to developers and the reduction in test quality that comes with a overly heavyweight harness. I do understand that Opera might have various requirements for what fits in their test harness, but I think if the goal is to create a harness for the web then we should optimize for what the web needs. Which is lots of tests and comprehensive ones. Another problem that I think we'd have is that many of our tests use generators and yield. This *dramatically* cuts down on the complexity of writing complex tests which has lots of asynchronous callbacks. For example [1][2] would have been much harder to write without them. I think our approach here could be migrate these tests to use ES6 based generators as soon as we have them implemented in gecko, and then submit them to W3C as soon as enough browsers implement ES6. I don't think that we should be telling people to not use generators in the meantime. My experience is that rewriting tests to use generators both cuts down on the test writing time, and makes it much less likely that the test ends up with intermittent orange bugs. If it's a pain to write a particular file in testharness.js, it can be kept as mochitest. In my experience, quite a lot of tests boil down to like ten lines, which would take about three minutes more to write using testharness.js than mochitest. Also, a test that's based on testharness.js but uses some Gecko-only features would be easier to make portable later than a test that's based on mochitest and also uses Gecko-only features. This doesn't match my experience at all. My experience is that writing tests has a high cost and results in fairly complex test files. / Jonas ___ dev-platform mailing
Re: Proposed policy change: reusability of tests by other browsers
On 10/9/12 6:46 PM, Jonas Sicking wrote: On Tue, Oct 9, 2012 at 2:43 AM, Aryeh Gregor a...@aryeh.name wrote: If it's a pain to write a particular file in testharness.js, it can be kept as mochitest. In my experience, quite a lot of tests boil down to like ten lines, which would take about three minutes more to write using testharness.js than mochitest. Also, a test that's based on testharness.js but uses some Gecko-only features would be easier to make portable later than a test that's based on mochitest and also uses Gecko-only features. This doesn't match my experience at all. My experience is that writing tests has a high cost and results in fairly complex test files. You're both right. Simple tests for really basic DOM stuff are very short. Tests for things like XHR and events and IndexedDB, which are a lot of what Jonas has had to write tests for are complete hell to write, if nothing else because of the async nature of those objects. We need a test harness that can handle both use cases without exploding. It's not quite clear to me that testharness.js is it, fwiw. -Boris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Tue, Oct 9, 2012 at 8:39 PM, Boris Zbarsky bzbar...@mit.edu wrote:
On 10/9/12 6:46 PM, Jonas Sicking wrote:
On Tue, Oct 9, 2012 at 2:43 AM, Aryeh Gregor a...@aryeh.name wrote:
If it's a pain to write a particular file in testharness.js, it can be
kept as mochitest. In my experience, quite a lot of tests boil down
to like ten lines, which would take about three minutes more to write
using testharness.js than mochitest. Also, a test that's based on
testharness.js but uses some Gecko-only features would be easier to
make portable later than a test that's based on mochitest and also
uses Gecko-only features.
This doesn't match my experience at all. My experience is that writing
tests has a high cost and results in fairly complex test files.
You're both right.
Simple tests for really basic DOM stuff are very short.
Tests for things like XHR and events and IndexedDB, which are a lot of
what Jonas has had to write tests for are complete hell to write, if
nothing else because of the async nature of those objects.
We need a test harness that can handle both use cases without exploding.
It's not quite clear to me that testharness.js is it, fwiw.
I'm a little confused, because I looked at testharness.js and thought it
looked perfectly fine, not particularly more or less complex than
SimpleTest/MochiTest.
Here's how I think you'd write a simple XHR test in both:
// SimpleTest aka MochiTest
req = new XMLHttpRequest();
req.open(GET, /example.json);
req.onreadystatechange = function () {
if (req.readyState != 4) {
return;
}
is(req.status, 200);
is(req.getResponseHeader(Content-Type), json);
SimpleTest.finish();
};
SimpleTest.waitForExplicitFinish();
// testharness
var t = async_test(Test XHR);
req = new XMLHttpRequest();
req.open(GET, /example.json);
req.onreadystatechange = function () {
if (req.readyState != 4) {
return;
}
t.step(function () {
assert_equals(req.status, 200);
assert_equals(req.getResponseHeader(Content-Type), json);
});
t.done();
};
Am I missing an important difference? Seems like testharness.js just wants
to add the concept of multiple tests on a page that can independently pass
or fail, and needs just a little more complexity as a result. But it
doesn't feel like a big deal.
(An aside from the topic of reusable tests, but I feel I should plug an
entirely different testing framework I've written: http://doctestjs.org –
which I think is especially good for spec testing, encouraging really
thorough testing and it makes test writing easy, including async tests.)
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 10/10/12 12:23 AM, Ian Bicking wrote: Here's how I think you'd write a simple XHR test in both: // SimpleTest aka MochiTest req = new XMLHttpRequest(); req.open(GET, /example.json); How did example.json get there? What if you need to test CORS? With mochitest at this point you're doing some SJS work and whatnot. // testharness And here you have to go and do whatever is appropriate to your server setup. Which is not part of testharness. Which is why the CORS tests imported from Opera to the W3C ended up all broken, because they did not configure the server correctly. -Boris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Tue, Oct 9, 2012 at 11:41 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 10/10/12 12:23 AM, Ian Bicking wrote: Here's how I think you'd write a simple XHR test in both: // SimpleTest aka MochiTest req = new XMLHttpRequest(); req.open(GET, /example.json); How did example.json get there? What if you need to test CORS? With mochitest at this point you're doing some SJS work and whatnot. // testharness And here you have to go and do whatever is appropriate to your server setup. Which is not part of testharness. Which is why the CORS tests imported from Opera to the W3C ended up all broken, because they did not configure the server correctly. OK – so if I understand the objection to testharness isn't anything in testharness.js itself, but that it's an incomplete solution as it doesn't define an environment? That is, MochiTest gives an environment where we can define resources at a variety of URLs and serve them with arbitrary headers, and so you can define tests that are more complete and self-contained. There's also stuff like permission overrides, which are really just about how MochiTest sets up the environment, and of course that stuff by its nature is not portable. But of course if you can't override permission checks it makes testing annoying. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
Sorry to bring back an old thread, but the upcoming Test the web
forward meeting reminded me of this thread.
In general I really approve of this idea, however I have one major concern.
2) Write an introduction to testharness.js targeted at people familiar
with mochitest. testharness.js is the de facto standard testing
harness in the web standards world, and we already can run such tests
as mochitests automatically (see dom/imptests/), so JavaScript tests
meant to be usable by other browsers should be written in that format.
As others have pointed out, the testharness.js test suite is much less
convenient to use than mochitest.
Simply wrapping
test(function() {
// test here
})
only works for the most simple tests. Most tests that I write use lots
of synchronous and asynchronous callbacks. Each one of those needs to
be wrapped to catch exceptions. There's also a lot more overhead in
the harness due to trying to count how much of a test you pass.
In general, testharness.js seems to be more optimized for producing a
result report which measure how close an implementation is to
implementing a feature, than it is optimized for making it easy to
write tests.
I believe many developers right now end up spending as much time
writing tests as they do implementing features. That is a very high
cost, but something that is definitely worth it. However we should be
working towards lowering that cost rather than increasing it.
Rather than trying to convince developers that testharness.js would in
fact not increase the cost of writing tests, I think we should try to
get W3C to adjust testharness.js such that it's easier to write tests
for it. If we make it as easy to write W3C tests as it is to write
mochitests then I would absolutely agree with your proposal. I would
imagine that would also make it easier to get other browser vendors to
do the same, as well as members of the webdevelopment community.
Another problem that I think we'd have is that many of our tests use
generators and yield. This *dramatically* cuts down on the complexity
of writing complex tests which has lots of asynchronous callbacks. For
example [1][2] would have been much harder to write without them. I
think our approach here could be migrate these tests to use ES6 based
generators as soon as we have them implemented in gecko, and then
submit them to W3C as soon as enough browsers implement ES6.
I don't think that we should be telling people to not use generators
in the meantime. My experience is that rewriting tests to use
generators both cuts down on the test writing time, and makes it much
less likely that the test ends up with intermittent orange bugs.
[1]
http://mxr.mozilla.org/mozilla-central/source/content/base/test/test_xhr_progressevents.html?force=1
[2]
http://mxr.mozilla.org/mozilla-central/source/dom/indexedDB/test/unit/test_add_put.js
/ Jonas
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 08/24/2012 11:08 PM, Brian Smith wrote: Aryeh Gregor wrote: 1) Decide on guidelines for whether a test is internal or reusable. As a starting point, I suggest that all tests that are regular webpages that don't use any Mozilla-specific features should be candidates for reuse. Examples of internal tests would be tests written in XUL and unit tests. In particular, I think we should write tests for reuse if they cover anything that other browsers implement or might implement, even if there's currently no standard for it. Other browsers should still be able to run these tests, even if they might decide not to follow them. Also, tests that currently use prefixed web-exposed properties should still be made reusable, since the properties should eventually be unprefixed. Which other browser makers are going to follow these guidelines, so that we benefit from them? Generally, this is a great idea if it makes it faster and easier to improve Firefox. But, like Asa, I also interpreted this proposal along the lines of Spend resources, and slow down Firefox development, to help other browsers. That seems totally in line with our values, but doesn't seem great as far as competitiveness is concerned. Looking at http://w3c-test.org/html/tests/submission/, there are tests from Apple, Google, Microsoft, and Opera, as well as from other organizations who benefit from interoperable implementations (Baidu, Comcast, Intel, …) and individuals (David Carlisle, Mathias Bynens, Philip Taylor, Aryeh, and myself). Some of those tests are already running on tinderbox. Also, are you saying if you are going to write a mochitest, then try to write a reusable test or if you are going to write a test, write a reusable test? The reason I ask is that we're supposed to write xpcshell tests in preference to mochitests when possible, and I'd hate the preference to change to be in favor of mochitests, because xpcshell tests are much more convenient (and faster) to write and run. Kyle already answered this. HTH Ms2ger ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
Aryeh Gregor wrote: 1) Decide on guidelines for whether a test is internal or reusable. As a starting point, I suggest that all tests that are regular webpages that don't use any Mozilla-specific features should be candidates for reuse. Examples of internal tests would be tests written in XUL and unit tests. In particular, I think we should write tests for reuse if they cover anything that other browsers implement or might implement, even if there's currently no standard for it. Other browsers should still be able to run these tests, even if they might decide not to follow them. Also, tests that currently use prefixed web-exposed properties should still be made reusable, since the properties should eventually be unprefixed. Which other browser makers are going to follow these guidelines, so that we benefit from them? Generally, this is a great idea if it makes it faster and easier to improve Firefox. But, like Asa, I also interpreted this proposal along the lines of Spend resources, and slow down Firefox development, to help other browsers. That seems totally in line with our values, but doesn't seem great as far as competitiveness is concerned. Also, are you saying if you are going to write a mochitest, then try to write a reusable test or if you are going to write a test, write a reusable test? The reason I ask is that we're supposed to write xpcshell tests in preference to mochitests when possible, and I'd hate the preference to change to be in favor of mochitests, because xpcshell tests are much more convenient (and faster) to write and run. Thanks, Brian ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Fri, Aug 24, 2012 at 2:08 PM, Brian Smith bsm...@mozilla.com wrote: Also, are you saying if you are going to write a mochitest, then try to write a reusable test or if you are going to write a test, write a reusable test? The reason I ask is that we're supposed to write xpcshell tests in preference to mochitests when possible, and I'd hate the preference to change to be in favor of mochitests, because xpcshell tests are much more convenient (and faster) to write and run. Most things that this is relevant to (things visible to web content) can't be tested from xpcshell anyways, so this shouldn't affect xpcshell vs. mochitests. - Kyle ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
Chris Hofmann schrieb: Yeah, this is not for the other browser vendors or users, but is mostly a benefit for web developers that want to count on certain behaviors to work across browsers effectively and reliably every release of every browser. And as web developers write websites for users, this is for users. And as we want to serve the web and users, it's a benefit for us. Robert Kaiser ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 20/08/12 18:25, Asa Dotzler wrote: Can you say more about this? Are you saying it's Mozilla's responsibility to put Mozilla resources into solving problems for Opera? I'm not sure I understand this assertion. I think he's arguing that a belief in user choice could well translate into doing things in a way which helps (or at least does not hinder) other browsers. How do you think our belief in user choice should translate into our attitude to other browsers using our work? Gerv ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 8/21/2012 6:34 AM, Gervase Markham wrote: On 20/08/12 18:25, Asa Dotzler wrote: Can you say more about this? Are you saying it's Mozilla's responsibility to put Mozilla resources into solving problems for Opera? I'm not sure I understand this assertion. I think he's arguing that a belief in user choice could well translate into doing things in a way which helps (or at least does not hinder) other browsers. How do you think our belief in user choice should translate into our attitude to other browsers using our work? Gerv I don't believe our belief in user choice should cause us to spend extra resources helping other browsers if those extra resources slow us down in our attempts to be effective competitors with those other browsers. - A ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Tuesday 2012-08-21 09:43 -0700, Asa Dotzler wrote: On 8/21/2012 6:34 AM, Gervase Markham wrote: On 20/08/12 18:25, Asa Dotzler wrote: Can you say more about this? Are you saying it's Mozilla's responsibility to put Mozilla resources into solving problems for Opera? I'm not sure I understand this assertion. I think he's arguing that a belief in user choice could well translate into doing things in a way which helps (or at least does not hinder) other browsers. How do you think our belief in user choice should translate into our attitude to other browsers using our work? Gerv I don't believe our belief in user choice should cause us to spend extra resources helping other browsers if those extra resources slow us down in our attempts to be effective competitors with those other browsers. In the long term, I think sharing tests with other browsers can be a speed-up, both for us and for the Web platform as a whole, since we: (a) have to spend less time changing our behavior when browsers are found to disagree with each other (something that's often more work when the problems are discovered later) (b) get to a more interoperable platform faster, which helps the Web compete against non-Web platforms -David -- 턞 L. David Baron http://dbaron.org/ 턂 턢 Mozilla http://www.mozilla.org/ 턂 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 8/21/2012 10:32 AM, L. David Baron wrote: On Tuesday 2012-08-21 09:43 -0700, Asa Dotzler wrote: On 8/21/2012 6:34 AM, Gervase Markham wrote: On 20/08/12 18:25, Asa Dotzler wrote: Can you say more about this? Are you saying it's Mozilla's responsibility to put Mozilla resources into solving problems for Opera? I'm not sure I understand this assertion. I think he's arguing that a belief in user choice could well translate into doing things in a way which helps (or at least does not hinder) other browsers. How do you think our belief in user choice should translate into our attitude to other browsers using our work? Gerv I don't believe our belief in user choice should cause us to spend extra resources helping other browsers if those extra resources slow us down in our attempts to be effective competitors with those other browsers. In the long term, I think sharing tests with other browsers can be a speed-up, both for us and for the Web platform as a whole, since we: (a) have to spend less time changing our behavior when browsers are found to disagree with each other (something that's often more work when the problems are discovered later) (b) get to a more interoperable platform faster, which helps the Web compete against non-Web platforms David, I can certainly see the value there. That is, IMO, quite different from the position I was responding to, Aryeh Gregor suggestion that our mission compells us to to put special effort into making things as easy as possible for smaller browsers. - A ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 12-08-18 9:42 AM, ja...@hoppipolla.co.uk wrote: On Friday, 17 August 2012 23:38:22 UTC+2, Justin Dolske wrote: I'm talking about the problem of having a large set of tests with a small percentage that fail intermittently, which is what we have today in m-c. Even if they all magically became cross-browser compatible right now, I think it could still be a tough sell to get other browsers vendors to run them. The rarity of a successful all-green run means you need people (like our tree sheriffs) to interpret what is a known failures and what is a real problem. AIUI Chromium has similar issues (any know about MS/Apple/Opera?), so if we were importing their tests we'd have to decide if that was worth it. I know about Opera ;) We have experienced the same kind of problems with randomly failing tests that you have, both in tests we have written ourselves and tests that have been imported from other places. We have put quite a lot of effort into fixing the problem and now have quite extensive systems for identifying unstable tests as soon as they are added to our test repository, and before they have the chance to cause the equivalent of intermittent orange. We also have ways to flag bogus changes in test status, and so can identify frequently misbehaving tests. As a consequence of this we have become better at writing stable tests, and by the time we release tests we are generally pretty confident that they are stable at least in Opera on our systems. We are also unafraid of using externally written tests because we can detect many quality issues before they have a chance to cost developers lots of time investigating bad results. Any program which relies on an event loop is by definition going to suffer from intermittent changes in behavior because of event ordering, etc. This means that some tests will fail intermittently in any web browser even if the browser itself is bug free, which means that any browser engine with a substantial amount of tests needs to come up with ways of dealing with those. Therefore, I agree with James that intermittently failing tests should not stop us from proceeding on sharing tests between different engines. Cheers, Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Tue, Aug 21, 2012 at 1:09 PM, Ehsan Akhgari ehsan.akhg...@gmail.comwrote: Any program which relies on an event loop is by definition going to suffer from intermittent changes in behavior because of event ordering, etc. This means that some tests will fail intermittently in any web browser even if the browser itself is bug free Not if the tests are written correctly! Of course, lots aren't, but A does not imply B here. - Kyle ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Tuesday, 21 August 2012 20:45:49 UTC+2, Asa Dotzler wrote: David, I can certainly see the value there. That is, IMO, quite different from the position I was responding to, Aryeh Gregor suggestion that our mission compells us to to put special effort into making things as easy as possible for smaller browsers. I think this is a side track. At least, when Opera release tests it is nothing to do with making life easy for other people; it's about improving interoperability between implementations, which is good for the platform and everyone who is invested in its success. Obviously to achieve this we do want as many people as possible to run the tests we write, and it does seem reasonable that you would want the same for tests that you release. I have already said that Opera will try to run as many tests as we can, and that we try to release tests we write when it makes sense. I believe you (Mozilla) can already run testharness.js tests (and reftests, obviously) and might be able to reuse some machinery that you have for importing CSSWG tests to import tests from other parts of the W3C. WebKit are also starting to write W3C-compatible tests in at least some cases, and are upgrading their test infrastructure to run these tests [1]. Microsoft also write testharness.js tests so I guess they must run them too although obviously it is rather harder to tell what's going on there. [1] http://krijnhoetmer.nl/irc-logs/whatwg/20120821#l-675 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 12-08-21 4:37 PM, Kyle Huey wrote: On Tue, Aug 21, 2012 at 1:09 PM, Ehsan Akhgari ehsan.akhg...@gmail.comwrote: Any program which relies on an event loop is by definition going to suffer from intermittent changes in behavior because of event ordering, etc. This means that some tests will fail intermittently in any web browser even if the browser itself is bug free Not if the tests are written correctly! Of course, lots aren't, but A does not imply B here. True. If both the software and the tests are correct, they will never fail by definition! But we know that neither of the two extremes is possible to attend to. Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 8/19/2012 1:41 AM, Aryeh Gregor wrote: Anyway, one major goal of an open web is that users should have as many choices as possible for web browsers. That means we need to put special effort into making things as easy as possible for smaller browsers. So if Opera will definitely use our tests and other browsers might or might not, I think that's a good reason to go ahead. Can you say more about this? Are you saying it's Mozilla's responsibility to put Mozilla resources into solving problems for Opera? I'm not sure I understand this assertion. - A ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Sat, Aug 18, 2012 at 12:38 AM, Justin Dolske dol...@mozilla.com wrote: Given the long history (shall I say plague?) of intermittent-orange in our tree, I can't agree that this would be a non-issue or is easy to fix! [Nor am I saying reusable tests are a bad idea -- just that it would seem wise to ramp up over time.] To be fair, the reason intermittent orange is such a headache for us is because our infrastructure for it is terrible. It all revolves around running tests in giant indivisible blocks that produce semi-formatted plaintext output, which is then parsed (using regex?) by various ad hoc tools, and huge amounts have to be done by hand. Random orange would be a drastically smaller problem for us if, e.g., we ran tests individually instead of in giant chunks, and automatically reran any failed test a few times to see if the failure is intermittent, and restarted a run in the middle if something made it crashed. Based on what James is saying, it sounds like Opera has a substantially more sophisticated system than we do (embarrassingly?). Anyway, one major goal of an open web is that users should have as many choices as possible for web browsers. That means we need to put special effort into making things as easy as possible for smaller browsers. So if Opera will definitely use our tests and other browsers might or might not, I think that's a good reason to go ahead. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 08/18/2012 12:08 AM, Gavin Sharp wrote: On Fri, Aug 17, 2012 at 5:38 PM, Justin Dolske dol...@mozilla.com wrote: I'm talking about the problem of having a large set of tests with a small percentage that fail intermittently, which is what we have today in m-c. What percentage of the intermittently-failing tests are tests for web features vs. tests for our UI or other Firefox-specific things, I wonder? It may be that the set of tests that would be shareable with other browsers is more reliable on average than the set of tests we have that are not relevant to other browsers. Also, the fact that a test fails intermittently in Gecko doesn't necessarily imply that it will also fail intermittently in other browsers; these failures often point to actual Gecko bugs, rather than test issues. HTH Ms2ger ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Friday, 17 August 2012 23:38:22 UTC+2, Justin Dolske wrote: I'm talking about the problem of having a large set of tests with a small percentage that fail intermittently, which is what we have today in m-c. Even if they all magically became cross-browser compatible right now, I think it could still be a tough sell to get other browsers vendors to run them. The rarity of a successful all-green run means you need people (like our tree sheriffs) to interpret what is a known failures and what is a real problem. AIUI Chromium has similar issues (any know about MS/Apple/Opera?), so if we were importing their tests we'd have to decide if that was worth it. I know about Opera ;) We have experienced the same kind of problems with randomly failing tests that you have, both in tests we have written ourselves and tests that have been imported from other places. We have put quite a lot of effort into fixing the problem and now have quite extensive systems for identifying unstable tests as soon as they are added to our test repository, and before they have the chance to cause the equivalent of intermittent orange. We also have ways to flag bogus changes in test status, and so can identify frequently misbehaving tests. As a consequence of this we have become better at writing stable tests, and by the time we release tests we are generally pretty confident that they are stable at least in Opera on our systems. We are also unafraid of using externally written tests because we can detect many quality issues before they have a chance to cost developers lots of time investigating bad results. Given the long history (shall I say plague?) of intermittent-orange in our tree, I can't agree that this would be a non-issue or is easy to fix! [Nor am I saying reusable tests are a bad idea -- just that it would seem wise to ramp up over time.] From our point of view it is no problem to import tests even if they are not 100% stable (and of course if the instability is due to a bug in Opera they could be stable for you and not for us). The worst case scenario is that we will simply disable the problematic tests but continue to get the benefit of all the other tests. That is a much better position to be in than not being able to run the tests at all. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Thu, Aug 16, 2012 at 6:10 PM, Ehsan Akhgari ehsan.akhg...@gmail.com wrote: 1. Is the current testharness.js API the documentation at the beginning of http://w3c-test.org/resources/testharness.js? If that is the case, the API looks a lot heavier weight than the default mochitest API we use. Not in practice. The assert_*() functions are pretty self-explanatory. They often make test code appreciably simpler or more rigorous. For instance, assert_throws() can check that the thing being thrown is a proper DOMException with all expected properties. In mochitests, it would be a pain to do every time, so often we'll just test that something is thrown but not test what it is, etc. Another advantage of having a lot of functions is that they produce nicer failure messages. Basically, as someone who's written lots of testharness.js tests and lots of mochitests: testharness.js is somewhat more complex, but not dramatically. If there's one big problem with shared tests, it's that we have to change the way we annotate expected failures. Currently we just go in and change ok() to todo() or whatever in the source code of the test, but of course that doesn't work for shared tests. testharness.js expects you to break things up into test()s of one or more assert_*()s, with the same number of test()s running no matter what, and each test() either passes or fails as a unit. Then you have to keep track of expected failures out-of-band (see files in dom/imptests/failures/). The major disadvantage of this is that if a test() tests multiple things and one of them is expected to fail, you lose regression-testing for any subsequent ones, because the test() aborts at the first assert failure. So in practice, it's not always clear where to divide up your test()s. One assert per test would be best for regression-testing, but it adds a lot of clutter to the source code. I think this is the one big thing that makes testharness.js more complicated to use than mochitest, although it's still not rocket science. If we decide test-per-assert is the way to go, perhaps we could get a series of functions added to make single-assert tests simpler. James, could you explain again what exactly the benefit is of this test/assert distinction? Mochitests effectively have one assert per test hardcoded in, and they work fine for us. 2. Is there any support for running reftest-style tests in a framework that is reusable by other browsers? If not, can we move to propose the reftest framework to the appropriate standards bodies so that it can be adopted by other browsers? Our reftest framework has been carefully designed to be Gecko-agnostic, and is much superior to the equivalent testing framework that WebKit has (not sure about other browser engines). Furthermore, the files loaded by this framework are not loaded in a privileged context with APIs such as SpecialPowers, which makes a large number of them portable to other browser engines. I think it makes sense for us if we can start this effort on the reftest framework, since that has a much lower barrier to entry, and ultimately this effort would be valuable only if other browser engines start to use our tests (and hopefully share theirs with us as well). The CSSWG has such a framework. Unfortunately, they're extremely demanding about accepting tests, requiring all kinds of documentation that it follows the standard, and they have formatting guidelines and so on and so forth. So it's not compatible with the idea of do things a little differently and everyone can use our tests. I agree that reftests would be easier to share, though. Crashtests would be even easier! But mochitests are really where most of our tests are. Also, unlike reftests, they can mostly be run in the browser with no special privileges. But as far as the actual sharing-tests thing goes, yes, it would make sense to start any kind of sharing initiative with crashtests, then reftests, then mochitests. On Thu, Aug 16, 2012 at 6:25 PM, Benjamin Smedberg benja...@smedbergs.us wrote: I agree with the first 3 points, but I object rather strongly to this one. I think we should try to keep the tests close to the relevant code whenever possible; this makes it more clear which module owner is responsible for the test, and makes it easier to find and run the relevant tests when modifying code. I think our system should try to keep this style of tests in the code module. There are two basic models you can have of test-sharing: 1) Everyone owns their own tests and just exports them to the world. Everyone else has to use them as-is or not at all; third parties can't make changes directly. This is compatible with whatever internal formatting we like. 2) Tests by all parties are put in a shared repository and maintained in common. Submitters don't own their tests; they're subject to review and adjustment by others. In this case, it doesn't make sense for us to organize the tests by our
Re: Proposed policy change: reusability of tests by other browsers
On 08/17/2012 01:22 PM, Aryeh Gregor wrote: If there's one big problem with shared tests, it's that we have to change the way we annotate expected failures. Currently we just go in and change ok() to todo() or whatever in the source code of the test, but of course that doesn't work for shared tests. testharness.js expects you to break things up into test()s of one or more assert_*()s, with the same number of test()s running no matter what, and each test() either passes or fails as a unit. Then you have to keep track of expected failures out-of-band (see files in dom/imptests/failures/). The major disadvantage of this is that if a test() tests multiple things and one of them is expected to fail, you lose regression-testing for any subsequent ones, because the test() aborts at the first assert failure. So in practice, it's not always clear where to divide up your test()s. One assert per test would be best for regression-testing, but it adds a lot of clutter to the source code. I think this is the one big thing that makes testharness.js more complicated to use than mochitest, although it's still not rocket science. If we decide test-per-assert is the way to go, perhaps we could get a series of functions added to make single-assert tests simpler. James, could you explain again what exactly the benefit is of this test/assert distinction? Mochitests effectively have one assert per test hardcoded in, and they work fine for us. So the theory is that a test can have preconditions that are needed for the rest of the test to make sense. For example if you are writing a history navigation test you might depend on a particular action adding an entry to the session history to get to the state where you can start your real test. If that fails you want the test to abort before it starts trying to manipulate the erroneous state. Hence the design that allows multiple asserts per logical test. I think for the way that Mozilla track regressions it works OK to just abort the whole file at the first sign of failure. That probably makes the Mochitest approach acceptable. however it doesn't work well for everyone. In particular it doesn't meet Opera's needs very well. I agree that reftests would be easier to share, though. Crashtests would be even easier! But mochitests are really where most of our tests are. Also, unlike reftests, they can mostly be run in the browser with no special privileges. But as far as the actual sharing-tests thing goes, yes, it would make sense to start any kind of sharing initiative with crashtests, then reftests, then mochitests. FWIW we already use some of your reftests. I think our import is rather out of date though, possibly dating from when we first implemented reftests and needed some to test the system. Our implementation is not 100% compatible with yours; for example we don't use reftest-wait but have special magic in our harness that achieves something similar in an entirely different way, but essentially the interoperability is very good. So if you put reftests somewhere it is easy for us to import them we will happily make use of them. On Thu, Aug 16, 2012 at 8:20 PM, L. David Baron dba...@dbaron.org wrote: It's two extra lines of boilerplate if you only have one test in the file. But if you have many tests in the file, it's a lot more, since each test needs to be wrapped in this -- at least in my understanding. Some browser vendors (e.g., Opera) seem to care quite strongly that each test file always execute the same number of tests in the same order -- even if some of those tests fail by throwing an exception. (We don't actually care about order. But we do care that the same test file always runs the same tests). The W3C already has test suites we can submit to in testharness.js format. We run some of those tests as mochitests; I know Opera does as well. I believe WebKit doesn't run them automatically yet. James Graham of Opera has indicated that they'd probably be interested in running our tests. (Opera gets much less user testing than we do, so they're very interested in automated testing.) Yes, we are interested in running your tests and very much hope that you are interested in running the tests that we write (we are trying, with some success, to share our testsuites with the W3C, although the process is not yet as slick as I would like). The relative amount of user testing various implementations get isn't really relevant; we like automated testing because it allows us to find bugs seconds to hours after the code is written rather than days or weeks. We like sharing tests because it improves the web platform as a whole by encouraging browsers to converge on interoperable behaviours. There are currently wild differences between UAs in some areas, and that doesn't benefit anyone. The basic idea here sounds worthy, but one concern is that our own tests are often unreliable in our own browser -- and I'd expect that
Re: Proposed policy change: reusability of tests by other browsers
On Fri, Aug 17, 2012 at 5:38 PM, Justin Dolske dol...@mozilla.com wrote: I'm talking about the problem of having a large set of tests with a small percentage that fail intermittently, which is what we have today in m-c. What percentage of the intermittently-failing tests are tests for web features vs. tests for our UI or other Firefox-specific things, I wonder? It may be that the set of tests that would be shareable with other browsers is more reliable on average than the set of tests we have that are not relevant to other browsers. Gavin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 8/16/12 12:07 PM, Ehsan Akhgari wrote: I agree with Benjamin here. In fact, I think if we take out item 4 completely Aryeh's proposal still makes sense. Where the tests live in our tree should not really matter. It matters insofar as it makes it more complicated to export our tests to the official W3C test suite, right? As long as we solve that problem in a locaton-agnostic way, we should be fine. -Boris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 08/16/2012 06:21 PM, Ehsan Akhgari wrote:
On 12-08-16 11:34 AM, Ms2ger wrote:
On 08/16/2012 05:10 PM, Ehsan Akhgari wrote:
I think this is generally a good idea. I have a few questions before
jumping in to agree though.
1. Is the current testharness.js API the documentation at the beginning
of http://w3c-test.org/resources/testharness.js? If that is the case,
the API looks a lot heavier weight than the default mochitest API we
use. In that case, would it make sense for us to have a compatibility
layer which translates our mochitest APIs to the equivalent
testharness.js API calls? (I'm not 100% sure if that conversion would
be straightforward.)
I don't feel it's terribly heavyweight. In the simple case, you need a
test(function() {
// tests
});
That's two extra lines of boilerplate. Our mochitest boilerplate is…
(*looks it up*) 30 lines. I think we'll deal. The method names for
assertions (assert_*) are a bit longer than we're used to, but in my
experience, you get used to it rather quickly.
I was not necessarily talking about the size of the boilerblate. I was
mostly talking about the size of the API. The mochitest API is
considerably smaller than testharness.js, and is therefore easier to
read and learn, I think.
I don't really agree that the API is much larger; it's just that all the
assert_* functions are documented, unlike the various functions we
expose to mochitests.
It's of course true that this is another API to learn, but I don't think
it's significantly harder than the mochitest API for people who don't
yet know either.
IMO, it would not be a good idea to try to layer another API on top of
this, because it makes our tests harder to understand and less reusable
by other browser vendors, and it means that experience with our brand of
testharness.js doesn't help much to understand the standard brand.
I would agree if the mochitest API was also huge. We're basically
talking about 3 functions: is, is_not and ok. I don't agree that using
these functions would make our tests any less usable by other browser
vendors. But I understand that in their eyes, these functions may not
be as intuitive as in mine. :-)
How about
SimpleTest.ise()/isa()/typeOf()/isDeeply()/waitForExplicitFinish()/finish()/executeSoon()/expectUncaughtException()/…?
I would argue that functions like assert_regexp_match don't get in your
way if you don't use them, just like SimpleTest.isDeeply() never gets in
my way when writing mochitests. In general, assert_{true,false},
assert_{equals,not_equals} and maybe assert_throws are, in my
experience, sufficient for most test.
I also don't think it would be terribly straightforward, and I think
that having real mochitests, testharness.js-that-look-like-mochitest and
standard testharness.js in our tree would make our testing situation
more confusing than the lessened learning curve would be worth. In
particular, having two APIs that superficially look the same but are
actually built on unrelated frameworks seems a recipe for a lot of
annoying differences in edge cases.
Hrm, which edge cases are you talking about? Let's talk about is(a, b,
msg) for example. Why do you think there would be cases where calling
is() is less understandable or more error prone than assert_equals?
Excellent example: is(undefined, null) passes, assert_equals(undefined,
null) fails.
However, I was thinking more about the fact that mochitest watches
window.onerror, while testharness.js catches exceptions from the
functions you pass to test(). To me, it seems easier to make sure you
use two different-looking APIs correctly, than to make sure you've got
the right API when one tries to disguise as the other.
On the testharness.js side, we have things like assert_regexp_match, for
example. I would argue that whether or not assert_regexp_match(a,
/foo/, msg) is more readable than ok(/foo/.match(a), msg) is very
subjective and depends on what the author of the test is used to see.
I wouldn't disagree.
HTH
Ms2ger
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Thursday 2012-08-16 17:34 +0200, Ms2ger wrote:
On 08/16/2012 05:10 PM, Ehsan Akhgari wrote:
I think this is generally a good idea. I have a few questions before
jumping in to agree though.
1. Is the current testharness.js API the documentation at the beginning
of http://w3c-test.org/resources/testharness.js? If that is the case,
the API looks a lot heavier weight than the default mochitest API we
use. In that case, would it make sense for us to have a compatibility
layer which translates our mochitest APIs to the equivalent
testharness.js API calls? (I'm not 100% sure if that conversion would
be straightforward.)
I don't feel it's terribly heavyweight. In the simple case, you need a
test(function() {
// tests
});
That's two extra lines of boilerplate. Our mochitest boilerplate is…
(*looks it up*) 30 lines. I think we'll deal. The method names for
assertions (assert_*) are a bit longer than we're used to, but in my
experience, you get used to it rather quickly.
It's two extra lines of boilerplate if you only have one test in the
file.
But if you have many tests in the file, it's a lot more, since each
test needs to be wrapped in this -- at least in my understanding.
Some browser vendors (e.g., Opera) seem to care quite strongly that
each test file always execute the same number of tests in the same
order -- even if some of those tests fail by throwing an exception.
So my understanding is that the intent here is that *each* test be
wrapped this way, presumably along with anything that might throw an
exception. (That said, I think this might throw concept is rather
loose.)
I think it's probably worth writing tests this way because of the
value of sharing them. But I wouldn't minimize that it is more
overhead.
One other characteristic of tests to be submitted to the W3C that's
rather important is that they fail when the feature isn't
implemented. If this isn't true, then people will build tables that
show a feature as being partially implemented, etc. (It's
particularly bad if, say, all but one of a large set of tests that
mostly test error handling actually pass when the feature isn't
implemented.)
-David
--
턞 L. David Baron http://dbaron.org/ 턂
턢 Mozilla http://www.mozilla.org/ 턂
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Thursday 2012-08-16 12:31 +0300, Aryeh Gregor wrote: I think that the above won't make anything much harder for our coders, but will be a big step forward for web testing -- especially if our example motivates other browsers to do the same. It needs a little I agree that this is worth doing. I think the key to making it work is figuring out how to distribute the knowledge effectively in the Mozilla community. This requires educating Mozilla module owners and code reviewers about the testing guidelines for W3C tests. Some of this can be done by written documentation, but some of it, I think, can only be taught through review and feedback cycles. In some cases, this means getting reasonably rapid feedback (from other browser vendors or others involved in W3C testing efforts) on submitted tests so that people at Mozilla who are writing tests can learn, through rapid feedback, what's required of tests submitted to W3C groups. That said, some test reviews in the W3C space tend to be unnecessarily nitpicky. I think we need to be careful to filter the review feedback appropriately for the change requests that are actually motivated by real testing needs, and to push back on the others so that the amount of information that we need to distribute through the Mozilla community is not too large. -David -- 턞 L. David Baron http://dbaron.org/ 턂 턢 Mozilla http://www.mozilla.org/ 턂 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 12-08-16 12:41 PM, Boris Zbarsky wrote: On 8/16/12 12:07 PM, Ehsan Akhgari wrote: I agree with Benjamin here. In fact, I think if we take out item 4 completely Aryeh's proposal still makes sense. Where the tests live in our tree should not really matter. It matters insofar as it makes it more complicated to export our tests to the official W3C test suite, right? As long as we solve that problem in a locaton-agnostic way, we should be fine. I would imagine having a manifest somewhere which points to the tests which can be submitted would solve that problem, right? Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
Ehsan Akhgari wrote: On the testharness.js side, we have things like assert_regexp_match, for example. I would argue that whether or not assert_regexp_match(a, /foo/, msg) is more readable than ok(/foo/.match(a), msg) is very subjective and depends on what the author of the test is used to see. I don't know whether we do a lot of regexp matching but if we did I would want it to use a dedicated method. I implemented ise because having an entire test reporting via ok(foo === bar, foo was not exactly equal to bar); does not make for good debugging. (Sadly I still wasn't able to work out why the test was failing and had to unCC myself from the bug again. (No it wasn't my test.)) -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Thursday, 16 August 2012 18:21:35 UTC+2, Ehsan Akhgari wrote: On the testharness.js side, we have things like assert_regexp_match, for example. I would argue that whether or not assert_regexp_match(a, /foo/, msg) is more readable than ok(/foo/.match(a), msg) is very subjective and depends on what the author of the test is used to see. As the original author of testharness.js this is one style choice I feel I can defend. The API is indeed relatively large – but no more so than the typical JUnit inspired framework – but this design has substantial benefits, and it is easy to ignore the parts that you aren't using. The typical test format we used at Opera before testharness.js was exceedingly lightweight/liberal; it basically has a single function that takes a boolean indicating whether the test passed. Sadly this makes the tests very inconsistent and hard to read because every one is a special snowflake that does its own validation logic. This requires lots of code, often duplicated across tests, that is not really related to the thing under test but is needed to validate the results. It also means that every author makes their own decisions about how to do various kinds of test e.g. checking the right exception was thrown by a DOM method. The advantage of having a richer API is that it brings rigor and consistency to tests. By far the most common assertion in testharness.js is assert_equals. Even this simple method is a win compared to having people write their own equality test because it will always use strict equality. For more complex cases where the logic needed to make the test correct is more involved the benefits are correspondingly greater e.g. assert_array_equals removes the need for an explicit loop cluttering up the test itself and assert_throws has some relatively complex logic to handle checking the expected properties of DOM and ECMAScript exceptions. If test authors were left to reimplement on an ad-hoc basis the assertions that testharness.js provides built-in tests would be harder to read, less consistent, and more prone to error. That seems well worth the cost of a having such a rich API. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On 8/16/12 8:10 AM, Ehsan Akhgari wrote: I think it makes sense for us if we can start this effort on the reftest framework, since that has a much lower barrier to entry, and ultimately this effort would be valuable only if other browser engines start to use our tests (and hopefully share theirs with us as well). Is there a concrete plan for getting other browsers to run these shared tests? The basic idea here sounds worthy, but one concern is that our own tests are often unreliable in our own browser -- and I'd expect that to only get worse as other browsers and their tests enter the picture. I'd therefore suggest that a successful cross-browser test effort should prioritize getting stuff running (even with just a handful of tests)... That way fun problems like reliability have a chance to be found/fixed over time, instead of having a megatestsuite suddenly appear that's unappealing to get working. Justin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
On Thu, Aug 16, 2012 at 2:39 PM, Justin Dolske dol...@mozilla.com wrote: On 8/16/12 8:10 AM, Ehsan Akhgari wrote: I think it makes sense for us if we can start this effort on the reftest framework, since that has a much lower barrier to entry, and ultimately this effort would be valuable only if other browser engines start to use our tests (and hopefully share theirs with us as well). Is there a concrete plan for getting other browsers to run these shared tests? The basic idea here sounds worthy, but one concern is that our own tests are often unreliable in our own browser -- and I'd expect that to only get worse as other browsers and their tests enter the picture. I'd therefore suggest that a successful cross-browser test effort should prioritize getting stuff running (even with just a handful of tests)... That way fun problems like reliability have a chance to be found/fixed over time, instead of having a megatestsuite suddenly appear that's unappealing to get working. That's not really true. Most of our mochitests and reftests are pretty solid. I don't think reliability will be a huge issue, and reliability problems due to bad tests can be fixed quite easily. - Kyle ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
