On Sat, Oct 6, 2012 at 6:25 AM, Jonas Sicking <jo...@sicking.cc> wrote: > In general, testharness.js seems to be more optimized for producing a > result report which measure how close an implementation is to > implementing a feature, than it is optimized for making it easy to > write tests.
I think it's actually optimized for Opera's internal testing needs as much as anything, since James Graham is the one who wrote it. Opera has a much more elaborate testing infrastructure than we do. I think they actually have stuff like databases instead of grepping plaintext output, and the ability to run individual test files, and automatically running new test files X times to record all intermittent failures. If we just layer it on top of mochitest, we can't take advantage of the greater structure, so the added complexity is wasted. James has indicated that features like "each file runs the same set of tests no matter how many fail" are essential to working properly with Opera's test setup. > I believe many developers right now end up spending as much time > writing tests as they do implementing features. That is a very high > cost, but something that is definitely worth it. However we should be > working towards lowering that cost rather than increasing it. > > Rather than trying to convince developers that testharness.js would in > fact not increase the cost of writing tests, I think we should try to > get W3C to adjust testharness.js such that it's easier to write tests > for it. If we make it as easy to write W3C tests as it is to write > mochitests then I would absolutely agree with your proposal. I would > imagine that would also make it easier to get other browser vendors to > do the same, as well as members of the webdevelopment community. I think testharness.js would definitely increase the costs of writing tests. I think the increase in interop would be worth it, if other browsers start using our tests. It's all good and well if we implement a feature correctly, but if other browsers implement it differently, it's not very useful to authors. This is why we want to submit tests to the W3C to start with. > Another problem that I think we'd have is that many of our tests use > generators and yield. This *dramatically* cuts down on the complexity > of writing complex tests which has lots of asynchronous callbacks. For > example [1][2] would have been much harder to write without them. I > think our approach here could be migrate these tests to use ES6 based > generators as soon as we have them implemented in gecko, and then > submit them to W3C as soon as enough browsers implement ES6. > > I don't think that we should be telling people to not use generators > in the meantime. My experience is that rewriting tests to use > generators both cuts down on the test writing time, and makes it much > less likely that the test ends up with intermittent orange bugs. If it's a pain to write a particular file in testharness.js, it can be kept as mochitest. In my experience, quite a lot of tests boil down to like ten lines, which would take about three minutes more to write using testharness.js than mochitest. Also, a test that's based on testharness.js but uses some Gecko-only features would be easier to make portable later than a test that's based on mochitest and also uses Gecko-only features. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
