Nice work, Jon. I've opened a task for defining a JJB builder/template and getting something like this into CI sooner rather than later.[1] I think your setup proves that a set of well-groomed MW-Selenium integration tests can be stable enough for this purpose, and we can start with an even smaller subset of core tests for a pre-merge build. Of course this isn't something that we planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so 'soon'—but we can start with an experiment on Gather or MobileFrontend tests, since their health has greatly improved, and see how it goes.
[1] https://phabricator.wikimedia.org/T103039 On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson <jdlrob...@gmail.com> wrote: > So the script that actually runs the browser test is not in a generic > useful form but it is: > https://gist.github.com/jdlrobson/32b607f8009e897ee80c > > It uses the GerritCommandLine tool to do grabbing and reviewing > https://github.com/jdlrobson/GerritCommandLine > > Ideally if we can use labs-tools-gerrit-to-redis for identifying > patches and then pulling them down we wouldn't need the > GerritCommandLine tool since the code to submit a review is pretty > trivial and captured in this function: > https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277 > > I've also put this in the task > https://phabricator.wikimedia.org/T101069#1379462 > > We'll probably want an instance per extension, to simplify having to > worry about dependencies (we can just run a git update on all > extensions after each checkout) > > On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez > <jhernan...@wikimedia.org> wrote: > > Awesome Jon! I'm so happy to finally see this developing :DD > > > > Loving the : `Browserbot happy!` > > > > I've noticed it can report either the name of the failing test or the > full > > log. What do you think if we show that, and a url with the pasted log > > somewhere publicly to not put too much noise on the comments but still be > > able to see it? Something like https://phabricator.wikimedia.org/paste/. > .. > > > > +1 to where is the source. > > +1 to documenting how you've set it all up on wiki somewhere. > > > > I also think we need a catchy phrase for the -1s! > > > > Thanks for you work on this, we'll get more focused time for it soon. > > > > On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith <samsm...@wikimedia.org> > wrote: > >> > >> I agree with Florian everything that you've written should be in a > public > >> version control system. > >> > >> Second, I'd ask that you document your experiences so far in getting > this > >> set up and how it works so that other members of the vertical can help > to > >> maintain it moving forward. > >> > >> Third, great work!!1 > >> > >> <3 > >> > >> –Sam > >> > >> On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.wel...@t-online.de > >> <florian.schmidt.wel...@t-online.de> wrote: > >>> > >>> > >>> > It's currently working via a script that you can find here: > >>> > /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh > >>> > >>> It would be great to have the script in a public version control system > >>> (e.g. github?), especially for people, e.g. volunteers, who can't ssh > to > >>> gather-browser-tests.eqiad.wmflabs[1] > >>> > >>> [1] all people, who're not members of > >>> https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests > >>> > >>> Best, > >>> Florian > >>> > >>> -----Original-Nachricht----- > >>> Betreff: [WikimediaMobile] [Update] Browser tests per patch > >>> Datum: Thu, 18 Jun 2015 03:27:32 +0200 > >>> Von: Jon Robson <jrob...@wikimedia.org> > >>> An: "QA (software quality assurance) for Wikimedia projects." > >>> <q...@lists.wikimedia.org>, mobile-l <mobile-l@lists.wikimedia.org> > >>> > >>> Background: mobile wants to gain more confidence in its browser tests > >>> by running a subset of browser tests on a case by case basis [0]. > >>> > >>> Good news: I've got a proof of concept running and Barry the browser > >>> test bot has given some legitimate helpful reviews to Gather [1]. > >>> > >>> Even better news: It's proving itself valuable already [2]. > >>> As you can see in the messages the bot has posted on [3] we have a > >>> couple of options on display option format for his reviews. > >>> > >>> So.. hopefully this short experience has sold you all already. > >>> > >>> This script is currently a manual job and needs a bit of tweaking > >>> before we can put it in a cron job/run it always - it needs to watch > >>> for new commits and then run a modification of the above script on a > >>> per case basis (if two versions of it run in parallel we have an > >>> issue). > >>> > >>> Definitely something we should push for next sprint! > >>> > >>> Long live Barry bot! > >>> > >>> Devs... (everyone else now of what follows is likely to be useful): > >>> I got the labs instance up and running on: > >>> http://gather-browser-tests.wmflabs.org/wiki/Main_Page > >>> > >>> Most of you in readership team should be able to ssh > >>> gather-browser-tests.eqiad.wmflabs > >>> Let me know if you have no access. > >>> > >>> It's currently working via a script that you can find here: > >>> /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh > >>> > >>> [0] https://phabricator.wikimedia.org/T100293 > >>> [1] > >>> > https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.com+status:open,n,z > >>> [2] https://gerrit.wikimedia.org/r/#/c/218731/ > >>> > >>> _______________________________________________ > >>> Mobile-l mailing list > >>> Mobile-l@lists.wikimedia.org > >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >>> > >>> > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> Mobile-l mailing list > >>> Mobile-l@lists.wikimedia.org > >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > >> > >> > >> _______________________________________________ > >> Mobile-l mailing list > >> Mobile-l@lists.wikimedia.org > >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > > > > > > _______________________________________________ > > Mobile-l mailing list > > Mobile-l@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > > > -- > Jon Robson > * http://jonrobson.me.uk > * https://www.facebook.com/jonrobson > * @rakugojon > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > -- Dan Duvall Automation Engineer Wikimedia Foundation <http://wikimediafoundation.org>
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
