On Thu, Jun 4, 2015 at 10:42 AM, Jon Robson <[email protected]> wrote:
> Cc'ing QA so they can see my latest reply. > > On Thu, Jun 4, 2015 at 10:42 AM, Jon Robson <[email protected]> wrote: > >> Gather tests take around 7 min 21s to run [1] >> Smoke tests 4 mins [2] >> These would be the initial target. >> > That seems like a more reasonable target albeit with proportionally less immediate payoff. Once we have the underlying CI isolation work done,[1] we should definitely shoot for pre-merge job for Gather and other repos with well-defined performant smoke tests. [1] https://phabricator.wikimedia.org/T47499 Yesterday 3 patches got merged that had been sitting in Gerrit for approx 3 >> days that caused the test to break in both Gather and MobileFrontend. >> Had we run the smoke tests on these tests at some point during those 3 >> days we would have been able to fix them pre-merging. >> > For the proposed setup to help in such cases, the job will need to be defined in JJB and scheduled to run via the gate-and-submit pipeline (pre-merge). At this point the job will be fully integrated into CI and will need to be quite fast and reliable lest it promote force merges. Personally I am fine with the delay and running the test jobs one at a >> time. I see this as a low risk useful grass routes experiment by a >> smaller team that can be fed up to the rest of the Foundation and benefit >> all teams. I don't see this as duplicating efforts rather than >> investigating something we know we all want but on a smaller scale. >> > Yes, but this isn't a grassroots effort. It's a time commitment from staff of WMF and we need to consider the cost (and technical debt) we're incurring. (Unless it is being done on volunteer time and, in that case, sorry for the misunderstanding.) I would rather see time commitments for figuring out how to fill the void in process that it seems we're lacking regarding test ownership and workflow around analyzing failures, filing tasks, and fixing broken tests. In other words, the relatively small amount of time you and I have spent over the past couple days has been relatively painless and yielded tangible payoff.[2] I'd like to see that kind of time investment be regular and explicitly promoted by leadership. [2] https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/707/ -- Dan Duvall Automation Engineer Wikimedia Foundation <http://wikimediafoundation.org>
_______________________________________________ QA mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/qa
