It strikes me that with devs writing tests there's a huge potential for gaming your system of only landing tests with bugs and/or patches.
Why would a dev want to write a rigid and reliable test? It delays the feature/bug fix, it's harder and it might have to be maintained in the future if it fails. We all know it's easy to write a test that's more likely to stay green over time. The reviewer on the other hand can take the green "OK to merge" of Travis far to literally and not thoroughly review the tests, particularly when under some time pressure. With a 2nd party writing tests and defining quality there's a balance to the equation, like the opposing sides of the stock market that, in the long term, keep things in equilibrium. ----- Original Message ----- From: Gareth Aye <gareth....@gmail.com> To: Zac Campbell <zcampb...@mozilla.com> Cc: Jonas Sicking <jo...@sicking.cc>, dev-g...@lists.mozilla.org, dev-b2g <dev-b2g@lists.mozilla.org> Sent: Tue, 19 Nov 2013 15:34:29 -0800 (PST) Subject: Re: [b2g] Let QA write your Gaia/Marionette UI tests! Zac - I didn't mean to offend! I got into this line of work by doing a lot of theoretical math, so if I use proof-by-exhaustion to point out an issue I have with your proposal, it doesn't mean that I don't want your help making firefox os a quality product. More test coverage is always good. I also know that b2g developers haven't always been great about writing tests. For feature/bug code that landed without tests, late is much better than never. However, I think I made some valid points about why (for future work/bugs/patches) we should strive to either have one person write the feature/bugfix and the tests or have dev/qa work land in the same pull request/patch. It's also true (and I think you sensed from my response) that I want developers to write and maintain tests. I believe this is the only scalable way forward. That doesn't mean that, for the test coverage we already lack, we couldn't use some help. On Tue, Nov 19, 2013 at 5:50 PM, Zac Campbell <zcampb...@mozilla.com> wrote: > Well geez i was just offering, kindly I thought, to have QA write some > additional test coverage, especially against real device hardware. I > thought it would be valuable. > > But I can read between the lines; you want Python and QA's tests to sod > off. > > > > ----- Original Message ----- > From: Gareth Aye <gareth....@gmail.com> > To: Zac Campbell <zcampb...@mozilla.com> > Cc: Jonas Sicking <jo...@sicking.cc>, dev-g...@lists.mozilla.org, dev-b2g > <dev-b2g@lists.mozilla.org> > Sent: Tue, 19 Nov 2013 14:28:43 -0800 (PST) > Subject: Re: [b2g] Let QA write your Gaia/Marionette UI tests! > > Ok well that's all well and good, but I think we should be careful about > how we do this. Will there be separate pull requests to Gaia? I posit that > the correct way to develop is to review and land test code alongside > feature/bug code. Let's prove this to ourselves by exploring other, wrong > ways in which we can do this. > > Case 1: Suppose we review/land our tests before the feature/bug code. > > - This breaks the build. > - It's possible that we learn things while writing the feature/bug code > about the constraints of the problem and that the feature changes. In this > case, our tests have gotten old and crufty before they ever passed! > - It makes our generous reviewers take two steps out of their busy days > instead of one. > > Case 2: Suppose we review/land our tests after the feature/bug code. > > - We check untested code into source control. Do we even know if it works? > - If the feature/bug code gets backed out and the test stays, then we break > the build. > - We waste our reviewer's time again making them review two patches. > > Our proof is finished since the only options are to land our test code > before, alongside, or after our bug code. Therefore, if we're going to have > people other than feature/bug code writers developing automated test cases, > then everyone who's working on the bug must work together to submit a > single patch and/or pull request to Gaia. This workflow isn't what I've > observed so far with the Python test code, and I hope that we keep to it > for the future health of our project. > > Personally this seems like a lot of hassle to me (which is why I prefer to > write my own tests), but I am happy for others to do this so long as they > do it thoughtfully (in a way that doesn't waste reviewer time, stop good > tests from being written, and break the build). > > > > On Tue, Nov 19, 2013 at 3:54 AM, Zac Campbell <zcampb...@mozilla.com> > wrote: > > > Well I was trying to offer you a QA/automation team to do that for you! > > > > but I am interested in new test cases for new functionality coming up > that > > QA haven't been let in on yet or if there's coverage that your team needs > > for on-device testing to use hardware that desktopb2g doesn't have access > > to. We can cover some space that Travis cannot. > > > > Julien has requested some for Messaging app that we did and Rudy also > > asked for a keyboard/TBPL test which we did last week. > > > > > > > > > > > > > > > > On 18/11/13 16:58, Gareth Aye wrote: > > > > I will let you, but only if you use the js tooling so that I can review > > and maintain the tests :) > > > > > > On Sat, Nov 16, 2013 at 3:47 PM, Jonas Sicking <jo...@sicking.cc> wrote: > > > >> On Nov 14, 2013 3:57 AM, "Zac Campbell" <zcampb...@mozilla.com> wrote: > >> > > >> > Are you tired? is the cat bothering you for its food? Forgotten to > shave > >> for the last 6 days? Don't have time to write your UI tests? > >> > > >> > Then let QA do it for you! > >> > >> Hahaha, I love it! Zac++ > >> > >> / Jonas > >> _______________________________________________ > >> dev-b2g mailing list > >> dev-b2g@lists.mozilla.org > >> https://lists.mozilla.org/listinfo/dev-b2g > >> > > > > > > > > -- > > Best, > > Gareth > > > > > > > > > -- > Best, > Gareth > > -- Best, Gareth _______________________________________________ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
