I have actually thought about this before! Most test suites need to start from a very "clean" slate with specific versions of the os and apps. Many test suites also make modifications to the phone state. Unless we backed up users' data, reinstalled the operating system, ran the tests, and then restored the users' data, I think it wouldn't work out. That sounds like a lot of effort and the potential to run into a lot of privacy/security issues. Not to say it's not worthwhile (perhaps it'd be more environmental to use our community's phones sometimes instead of purchasing specialized hardware for our tests), but we should definitely weigh the costs and benefits since there might be a lot of complexity/engineering under the hood.
On Thu, Jun 5, 2014 at 5:20 PM, Natalia Martinez-Winter <nata...@mozilla.com > wrote: > stop me if I'm totally out of scope... > would it make sense in the future to have a (peer-to-peer ?) solution to > run tests on devices across the world (especially from the community, tests > running at night when devices are not being used) ? > > > On Tue Jun 3 20:23:39 2014, Jonathan Griffin wrote: > >> Hi all, >> >> There have been a couple of threads related to test automation in B2G, >> asking why we haven't caught some especially egregious regressions; >> the kind that basically "break the phone". >> >> To answer that, I'd like to describe how our on-device automation >> currently works, and what we're doing to expand it so we can more >> effectively address these concerns. >> >> We currently have a smallish number of real devices, managed by WebQA, >> hooked up to on-device automation. They run a bank of tests against a >> number of branches several times a day. The devices are >> time-consuming to manage, since they occasionally get wedged during >> flashing, rebooting, or other operations, and require manual >> intervention to fix. For this reason, and because it's been very >> difficult to obtain significant numbers of devices for automation, we >> haven't been able to run any tests frequently enough to provide >> per-commit coverage. >> >> When tests fail, WebQA engages in a fairly time-consuming process of >> investigation and bisection. In the case of the homescreen breakage >> (caused by https://bugzilla.mozilla.org/show_bug.cgi?id=957086), our >> on-device tests did break, and the team was in the process of >> investigating these failures, which has to be done in order to be able >> to create specific, actionable bugs. >> >> Clearly, what we really want is to be able to run at least a small set >> of tests per-commit, so that when things break, we don't need to spend >> lots of time investigating...we will already know which commit caused >> the problem, and can back it out or address it otherwise promptly. >> >> That's exactly what we are planning for Q3, thanks to the Flame >> device. Jonathan Hylands has developed a power harness for this that >> allows us to remotely restart the phone, which addresses some of the >> device management concerns. The A*Team, WebQA, and jhylands are >> working together to get 30 Flames in automation, and to reduce their >> management costs. This is enough to allow us to run a small set of >> functional and performance tests per-commit, which should be enough to >> catch most "break the phone" problems. >> >> Another issue we've had with device testing is test result visibility; >> currently, test results are available on Jenkins, for which you need >> VPN access. This is awkward for people not closely involved in >> maintaining and running the tests. >> >> Next quarter, we will be improving this as well. Jonathan Eads on the >> A*Team is currently in the process of deploying Treeherder, a >> successor to TBPL. Unlike TBPL, Treeherder is not tightly coupled >> with buildbot, and is capable of displaying test results from >> arbitrary data sources. As our bank of 30 Flames becomes available, we >> will start publishing on-device test results to Treeherder, in the >> same UI that will be used to display the per-commit tests being run in >> buildbot. This will give people a "one-stop shop" for seeing test >> results for B2G, regardless of whether they're run on devices or in >> VM's managed by buildbot. >> >> Both of these pieces together will give us the ability to manage some >> on-device tests in a manner similar to the way we currently handle >> desktop and emulator tests in TBPL; especially bad commits should >> break tests, the breakage should be visible in Treeherder, and the >> sheriffs will back out the offending commits. >> >> We won't have enough device capacity to run all device tests >> per-commit, at least at first. We'll have to carefully select a small >> set of tests that guard against the worst kinds of breakage. Whether >> we can scale beyond 30 devices will depend on how stable the devices >> are and what their management costs are, which is something we'll be >> looking at over the next few months. >> >> Regards, >> >> Jonathan >> >> _______________________________________________ >> dev-b2g mailing list >> dev-b2g@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-b2g >> > > -- > Natalia Martinez-Winter > > Channel Marketing Manager > Firefox OS > +33 6 88811959 > nataliamozi...@skype.com > @martinezwinter > > _______________________________________________ > dev-b2g mailing list > dev-b2g@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-b2g >
_______________________________________________ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
