Re: Proposed policy change: reusability of tests by other browsers

On 11/12/2012 9:53 PM, Jeff Walden wrote: > At the time the web server was introduced I don't believe we had > Python as a build requirement, so we couldn't have used some > Python-based server (the option most likely to be somehow portable > across browsers/engines). That probably could be address

Re: Proposed policy change: reusability of tests by other browsers

I read newsgroups too little (or perhaps just enough, or too much, depending), sorry for the kind-of-late response here... On 10/10/2012 01:26 AM, ja...@hoppipolla.co.uk wrote: > Do you have a concrete suggestion for how to improve this, in a way that > works cross-browser? Well, the cross-doma

Re: Proposed policy change: reusability of tests by other browsers

On 11/08/2012 01:10 PM, Aryeh Gregor wrote: On Wed, Nov 7, 2012 at 4:13 PM, James Graham wrote: There is an experimental branch with this mode in; it isn't production quality yet. I am still unsure that it's a good idea; in particular I think it encourages people to write multiple tests on the

Re: Proposed policy change: reusability of tests by other browsers

On Sun, Nov 11, 2012 at 4:21 PM, Neil wrote: > Why not simply verify that the list of actual fails equals the list of > expected fails, and report items that are only in one of the two lists? That would be a bit more robust, yes, and it should be doable without much work. It still wouldn't detec

Re: Proposed policy change: reusability of tests by other browsers

Aryeh Gregor wrote: Currently we only check that no test fails that's not on the per-file whitelist of expected fails, and in practice that works fine for us. If we wanted to be pickier, we could list all expected results, both pass and fail, and verify that the lists exactly match. This is

Re: Proposed policy change: reusability of tests by other browsers

On Sat, Nov 10, 2012 at 9:41 AM, Boris Zbarsky wrote: > I believe right now we have a list of "known failures" alongside such tests, > and our own test harness knows to compare what the tests are reporting to > our list of known failures. As in, we're not using the pass/fail state of > the tests

Re: Proposed policy change: reusability of tests by other browsers

On 11/9/12 12:52 AM, James Graham wrote: I know Mozilla use a system where all the tests in a file should pass, but I don't see how that will work well when you don't control the tests. If you are manually editing every file when you import it, I imagine that updating tests will be so time consum

Re: Proposed policy change: reusability of tests by other browsers

On 11/07/2012 02:03 PM, Henri Sivonen wrote: On Wed, Oct 10, 2012 at 10:46 AM, Aryeh Gregor wrote: That said, of course, Mozilla hackers *are* familiar with Mochitest but not testharness.js, and adopting testharness.js in parallel with Mochitest would require people to be familiar with both. T

Re: Proposed policy change: reusability of tests by other browsers

On Wed, Nov 7, 2012 at 4:13 PM, James Graham wrote: > There is an experimental branch with this mode in; it isn't production > quality yet. I am still unsure that it's a good idea; in particular I think > it encourages people to write multiple tests on the same page in such a way > that if one fai

Re: Proposed policy change: reusability of tests by other browsers

On Wed, Oct 10, 2012 at 10:46 AM, Aryeh Gregor wrote: > That said, of course, Mozilla hackers *are* familiar with Mochitest > but not testharness.js, and adopting testharness.js in parallel with > Mochitest would require people to be familiar with both. That is > certainly a minus. I was told at

Re: Proposed policy change: reusability of tests by other browsers

On Wednesday, 10 October 2012 07:18:45 UTC+2, Boris Zbarsky wrote: > On 10/10/12 1:13 AM, Ian Bicking wrote: > > OK – so if I understand the objection to testharness isn't anything in > > > testharness.js itself, but that it's an incomplete solution as it doesn't > > define an environment? > > T

Re: Proposed policy change: reusability of tests by other browsers

On Wed, Oct 10, 2012 at 12:46 AM, Jonas Sicking wrote: > However, for someone working against a looming deadline, the cost of > writing sharable tests and risk missing the deadline can be much > higher than having those tests not be run by other browser vendors. Which is why we have rolling relea

Re: Proposed policy change: reusability of tests by other browsers

On 10/10/12 1:13 AM, Ian Bicking wrote: OK – so if I understand the objection to testharness isn't anything in testharness.js itself, but that it's an incomplete solution as it doesn't define an environment? That's _my_ primary objection, after looking at it briefly and seeing how it works in

Re: Proposed policy change: reusability of tests by other browsers

On Tue, Oct 9, 2012 at 11:41 PM, Boris Zbarsky wrote: > On 10/10/12 12:23 AM, Ian Bicking wrote: > >> Here's how I think you'd write a simple XHR test in both: >> >> // SimpleTest aka MochiTest >> req = new XMLHttpRequest(); >> req.open("GET", "/example.json"); >> > > How did example.json get the

Re: Proposed policy change: reusability of tests by other browsers

On 10/10/12 12:23 AM, Ian Bicking wrote: Here's how I think you'd write a simple XHR test in both: // SimpleTest aka MochiTest req = new XMLHttpRequest(); req.open("GET", "/example.json"); How did example.json get there? What if you need to test CORS? With mochitest at this point you're doin

Re: Proposed policy change: reusability of tests by other browsers

On Tue, Oct 9, 2012 at 8:39 PM, Boris Zbarsky wrote: > On 10/9/12 6:46 PM, Jonas Sicking wrote: > >> On Tue, Oct 9, 2012 at 2:43 AM, Aryeh Gregor wrote: >> >>> If it's a pain to write a particular file in testharness.js, it can be >>> kept as mochitest. In my experience, quite a lot of tests bo

Re: Proposed policy change: reusability of tests by other browsers

On 10/9/12 6:46 PM, Jonas Sicking wrote: On Tue, Oct 9, 2012 at 2:43 AM, Aryeh Gregor wrote: If it's a pain to write a particular file in testharness.js, it can be kept as mochitest. In my experience, quite a lot of tests boil down to like ten lines, which would take about three minutes more t

Re: Proposed policy change: reusability of tests by other browsers

On Tue, Oct 9, 2012 at 2:43 AM, Aryeh Gregor wrote: > On Sat, Oct 6, 2012 at 6:25 AM, Jonas Sicking wrote: >> In general, testharness.js seems to be more optimized for producing a >> result report which measure how close an implementation is to >> implementing a feature, than it is optimized for

Re: Proposed policy change: reusability of tests by other browsers

On Sat, Oct 6, 2012 at 6:25 AM, Jonas Sicking wrote: > In general, testharness.js seems to be more optimized for producing a > result report which measure how close an implementation is to > implementing a feature, than it is optimized for making it easy to > write tests. I think it's actually op

Re: Proposed policy change: reusability of tests by other browsers

Sorry to bring back an old thread, but the upcoming "Test the web forward" meeting reminded me of this thread. In general I really approve of this idea, however I have one major concern. > 2) Write an introduction to testharness.js targeted at people familiar > with mochitest. testharness.js is

Re: Proposed policy change: reusability of tests by other browsers

On 08/24/2012 11:08 PM, Brian Smith wrote: Aryeh Gregor wrote: 1) Decide on guidelines for whether a test is internal or reusable. As a starting point, I suggest that all tests that are regular webpages that don't use any Mozilla-specific features should be candidates for reuse. Examples of int

Re: Proposed policy change: reusability of tests by other browsers

On Fri, Aug 24, 2012 at 2:08 PM, Brian Smith wrote: > Also, are you saying "if you are going to write a mochitest, then try to > write a reusable test" or "if you are going to write a test, write a > reusable test?" The reason I ask is that we're supposed to write xpcshell > tests in preference t

Re: Proposed policy change: reusability of tests by other browsers

Aryeh Gregor wrote: > 1) Decide on guidelines for whether a test is internal or reusable. > As a starting point, I suggest that all tests that are regular > webpages that don't use any Mozilla-specific features should be > candidates for reuse. Examples of internal tests would be tests > written i

Re: Proposed policy change: reusability of tests by other browsers

Chris Hofmann schrieb: Yeah, this is not for the other browser vendors or users, but is mostly a benefit for web developers that want to count on certain behaviors to work across browsers effectively and reliably every release of every browser. And as web developers write websites for users, th

Re: Proposed policy change: reusability of tests by other browsers

On 12-08-21 4:37 PM, Kyle Huey wrote: On Tue, Aug 21, 2012 at 1:09 PM, Ehsan Akhgari wrote: Any program which relies on an event loop is by definition going to suffer from intermittent changes in behavior because of event ordering, etc. This means that some tests will fail intermittently in any

Re: Proposed policy change: reusability of tests by other browsers

On 8/21/12 10:32 AM, L. David Baron wrote: On Tuesday 2012-08-21 09:43 -0700, Asa Dotzler wrote: On 8/21/2012 6:34 AM, Gervase Markham wrote: On 20/08/12 18:25, Asa Dotzler wrote: Can you say more about this? Are you saying it's Mozilla's responsibility to put Mozilla resources into solving pr

Re: Proposed policy change: reusability of tests by other browsers

On Tuesday, 21 August 2012 20:45:49 UTC+2, Asa Dotzler wrote: > David, I can certainly see the value there. That is, IMO, quite > different from the position I was responding to, Aryeh Gregor suggestion > that our mission compells us to "to put special effort into making > things as easy as po

Re: Proposed policy change: reusability of tests by other browsers

On Tue, Aug 21, 2012 at 1:09 PM, Ehsan Akhgari wrote: > Any program which relies on an event loop is by definition going to suffer > from intermittent changes in behavior because of event ordering, etc. This > means that some tests will fail intermittently in any web browser even if > the browser

Re: Proposed policy change: reusability of tests by other browsers

On 12-08-18 9:42 AM, ja...@hoppipolla.co.uk wrote: On Friday, 17 August 2012 23:38:22 UTC+2, Justin Dolske wrote: I'm talking about the problem of having a large set of tests with a small percentage that fail intermittently, which is what we have today in m-c. Even if they all magically became

Re: Proposed policy change: reusability of tests by other browsers

On 8/21/2012 10:32 AM, L. David Baron wrote: On Tuesday 2012-08-21 09:43 -0700, Asa Dotzler wrote: On 8/21/2012 6:34 AM, Gervase Markham wrote: On 20/08/12 18:25, Asa Dotzler wrote: Can you say more about this? Are you saying it's Mozilla's responsibility to put Mozilla resources into solving

Re: Proposed policy change: reusability of tests by other browsers

On Tuesday 2012-08-21 09:43 -0700, Asa Dotzler wrote: > On 8/21/2012 6:34 AM, Gervase Markham wrote: > >On 20/08/12 18:25, Asa Dotzler wrote: > >>Can you say more about this? Are you saying it's Mozilla's > >>responsibility to put Mozilla resources into solving problems for Opera? > >>I'm not sure

Re: Proposed policy change: reusability of tests by other browsers

On 8/21/2012 6:34 AM, Gervase Markham wrote: On 20/08/12 18:25, Asa Dotzler wrote: Can you say more about this? Are you saying it's Mozilla's responsibility to put Mozilla resources into solving problems for Opera? I'm not sure I understand this assertion. I think he's arguing that a belief in

Re: Proposed policy change: reusability of tests by other browsers

On 20/08/12 18:25, Asa Dotzler wrote: > Can you say more about this? Are you saying it's Mozilla's > responsibility to put Mozilla resources into solving problems for Opera? > I'm not sure I understand this assertion. I think he's arguing that a belief in user choice could well translate into doin

Re: Proposed policy change: reusability of tests by other browsers

On 8/19/2012 1:41 AM, Aryeh Gregor wrote: Anyway, one major goal of an open web is that users should have as many choices as possible for web browsers. That means we need to put special effort into making things as easy as possible for smaller browsers. So if Opera will definitely use our test

Re: Proposed policy change: reusability of tests by other browsers

On Sat, Aug 18, 2012 at 12:38 AM, Justin Dolske wrote: > Given the long history (shall I say "plague"?) of intermittent-orange in our > tree, I can't agree that this would be a non-issue or is easy to fix! [Nor > am I saying reusable tests are a bad idea -- just that it would seem wise to > ramp u

Re: Proposed policy change: reusability of tests by other browsers

On Friday, 17 August 2012 23:38:22 UTC+2, Justin Dolske wrote: > I'm talking about the problem of having a large set of tests with a > small percentage that fail intermittently, which is what we have today > in m-c. Even if they all magically became cross-browser compatible right > now, I thin

Re: Proposed policy change: reusability of tests by other browsers

On 08/18/2012 12:08 AM, Gavin Sharp wrote: On Fri, Aug 17, 2012 at 5:38 PM, Justin Dolske wrote: I'm talking about the problem of having a large set of tests with a small percentage that fail intermittently, which is what we have today in m-c. What percentage of the intermittently-failing tes

Re: Proposed policy change: reusability of tests by other browsers

On Fri, Aug 17, 2012 at 5:38 PM, Justin Dolske wrote: > I'm talking about the problem of having a large set of tests with a small > percentage that fail intermittently, which is what we have today in m-c. What percentage of the intermittently-failing tests are tests for web features vs. tests for

Re: Proposed policy change: reusability of tests by other browsers

On 8/16/12 3:01 PM, Kyle Huey wrote: The basic idea here sounds worthy, but one concern is that our own tests are often unreliable in our own browser -- and I'd expect that to only get worse as other browsers and their tests enter the picture. That's not really true. Most of our mochitests an

Re: Proposed policy change: reusability of tests by other browsers

On 08/17/2012 01:22 PM, Aryeh Gregor wrote: If there's one big problem with shared tests, it's that we have to change the way we annotate expected failures. Currently we just go in and change ok() to todo() or whatever in the source code of the test, but of course that doesn't work for shared t

Re: Proposed policy change: reusability of tests by other browsers

On Thu, Aug 16, 2012 at 6:10 PM, Ehsan Akhgari wrote: > 1. Is the current testharness.js API the documentation at the beginning of > ? If that is the case, the > API looks a lot heavier weight than the default mochitest API we use. Not in practice.

Re: Proposed policy change: reusability of tests by other browsers

On Thu, Aug 16, 2012 at 2:39 PM, Justin Dolske wrote: > On 8/16/12 8:10 AM, Ehsan Akhgari wrote: > > I think it makes sense for us if we can start this effort on the reftest >> framework, since that has a much lower barrier to entry, and ultimately >> this effort would be valuable only if other

Re: Proposed policy change: reusability of tests by other browsers

On 8/16/12 8:10 AM, Ehsan Akhgari wrote: I think it makes sense for us if we can start this effort on the reftest framework, since that has a much lower barrier to entry, and ultimately this effort would be valuable only if other browser engines start to use our tests (and hopefully share theirs

Re: Proposed policy change: reusability of tests by other browsers

On Thursday, 16 August 2012 18:21:35 UTC+2, Ehsan Akhgari wrote: > On the testharness.js side, we have things like assert_regexp_match, for > > example. I would argue that whether or not assert_regexp_match(a, > > /foo/, "msg") is more readable than ok(/foo/.match(a), "msg") is very > > sub

Re: Proposed policy change: reusability of tests by other browsers

Ehsan Akhgari wrote: On the testharness.js side, we have things like assert_regexp_match, for example. I would argue that whether or not assert_regexp_match(a, /foo/, "msg") is more readable than ok(/foo/.match(a), "msg") is very subjective and depends on what the author of the test is used t

Re: Proposed policy change: reusability of tests by other browsers

On 8/16/12 3:38 PM, Ehsan Akhgari wrote: I would imagine having a manifest somewhere which points to the tests which can be submitted would solve that problem, right? Sure. Just need to maintain that manifest as new tests (or just new test dirs?) are added. -Boris _

Re: Proposed policy change: reusability of tests by other browsers

On 12-08-16 12:41 PM, Boris Zbarsky wrote: On 8/16/12 12:07 PM, Ehsan Akhgari wrote: I agree with Benjamin here. In fact, I think if we take out item 4 completely Aryeh's proposal still makes sense. Where the tests live in our tree should not really matter. It matters insofar as it makes it

Re: Proposed policy change: reusability of tests by other browsers

On Thursday 2012-08-16 12:31 +0300, Aryeh Gregor wrote: > I think that the above won't make anything much harder for our coders, > but will be a big step forward for web testing -- especially if our > example motivates other browsers to do the same. It needs a little I agree that this is worth do

Re: Proposed policy change: reusability of tests by other browsers

On Thursday 2012-08-16 17:34 +0200, Ms2ger wrote: > On 08/16/2012 05:10 PM, Ehsan Akhgari wrote: > >I think this is generally a good idea. I have a few questions before > >jumping in to agree though. > > > >1. Is the current testharness.js API the documentation at the beginning > >of

Re: Proposed policy change: reusability of tests by other browsers

On 08/16/2012 06:21 PM, Ehsan Akhgari wrote: On 12-08-16 11:34 AM, Ms2ger wrote: On 08/16/2012 05:10 PM, Ehsan Akhgari wrote: I think this is generally a good idea. I have a few questions before jumping in to agree though. 1. Is the current testharness.js API the documentation at the beginnin

Re: Proposed policy change: reusability of tests by other browsers

On 8/16/12 12:07 PM, Ehsan Akhgari wrote: I agree with Benjamin here. In fact, I think if we take out item 4 completely Aryeh's proposal still makes sense. Where the tests live in our tree should not really matter. It matters insofar as it makes it more complicated to export our tests to the

Re: Proposed policy change: reusability of tests by other browsers

On 12-08-16 11:34 AM, Ms2ger wrote: On 08/16/2012 05:10 PM, Ehsan Akhgari wrote: I think this is generally a good idea. I have a few questions before jumping in to agree though. 1. Is the current testharness.js API the documentation at the beginning of

Re: Proposed policy change: reusability of tests by other browsers

On 12-08-16 11:25 AM, Benjamin Smedberg wrote: On 8/16/2012 5:31 AM, Aryeh Gregor wrote: 4) Require that all new tests that qualify as reusable must be checked into a specific new directory created for this purpose, rather than someplace near the code as they are currently. Reviewers need to

Re: Proposed policy change: reusability of tests by other browsers

On 08/16/2012 05:10 PM, Ehsan Akhgari wrote: I think this is generally a good idea. I have a few questions before jumping in to agree though. 1. Is the current testharness.js API the documentation at the beginning of ? If that is the case, the API

Re: Proposed policy change: reusability of tests by other browsers

On 8/16/2012 5:31 AM, Aryeh Gregor wrote: 4) Require that all new tests that qualify as reusable must be checked into a specific new directory created for this purpose, rather than someplace near the code as they are currently. Reviewers need to eventually start giving r- for tests written in

Re: Proposed policy change: reusability of tests by other browsers

I think this is generally a good idea. I have a few questions before jumping in to agree though. 1. Is the current testharness.js API the documentation at the beginning of ? If that is the case, the API looks a lot heavier weight than the default

Proposed policy change: reusability of tests by other browsers

Mozilla has a long-standing policy that with certain limited exceptions, all code changes must be accompanied by a test. Following this policy has given us an excellent and steadily growing regression test-suite. Some of these tests are very specific to Mozilla, but a substantial fraction test ou