Baha I agree with you, but instead of staying silent just commenting a "BarryBot is happy! Browser tests passed" without setting any +1 would be good to know they run.
I also agree with Florian, if we could set Verified +1 or -1 that could be interesting. How does it affect jenkins if something/somebody sets verified to -1 for example? Would it block it from merging? On Wed, Jun 24, 2015 at 1:53 PM, florian.schmidt.wel...@t-online.de < florian.schmidt.wel...@t-online.de> wrote: > Or, maybe, it should be possible for the bot to set the verified flag, > instead of the code review (it's not a code review, so a minus one is > misleading, too). > > > > Freundliche Grüße > Florian Schmidt > > > > > > > > -----Original-Nachricht----- > > Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch > > Datum: Wed, 24 Jun 2015 13:47:39 +0200 > > Von: Bahodir Mansurov <bmansu...@wikimedia.org> > > An: Jon Robson <jdlrob...@gmail.com> > > > > > > > May I suggest that Barry should not +1 patches. It gives a false > impression that the code is alright even though the code may not be covered > at all. Can we have it just -1 when there is a problem, and stay silent > otherwise? > > > On Jun 18, 2015, at 8:55 PM, Jon Robson <jdlrob...@gmail.com> wrote: > > So far I'm seeing some extremely positive results. The tests are > running super fast (in phantomjs the smoke tests are taking less than > 3 minutes...). > > I've been manually running him whilst I do code review in parallel and > he's already generated some interesting conversation on this patchset: > https://gerrit.wikimedia.org/r/#/c/219249/ > > If you want to pair and get this to be less hacky I'm more than happy! > > > On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall <dduv...@wikimedia.org> wrote: > > Nice work, Jon. > > I've opened a task for defining a JJB builder/template and getting > something > like this into CI sooner rather than later.[1] I think your setup proves > that a set of well-groomed MW-Selenium integration tests can be stable > enough for this purpose, and we can start with an even smaller subset of > core tests for a pre-merge build. Of course this isn't something that we > planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so > 'soon'—but we can start with an experiment on Gather or MobileFrontend > tests, since their health has greatly improved, and see how it goes. > > [1] https://phabricator.wikimedia.org/T103039 > > On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson <jdlrob...@gmail.com> wrote: > > > So the script that actually runs the browser test is not in a generic > useful form but it is: > https://gist.github.com/jdlrobson/32b607f8009e897ee80c > > It uses the GerritCommandLine tool to do grabbing and reviewing > https://github.com/jdlrobson/GerritCommandLine > > Ideally if we can use labs-tools-gerrit-to-redis for identifying > patches and then pulling them down we wouldn't need the > GerritCommandLine tool since the code to submit a review is pretty > trivial and captured in this function: > https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277 > > I've also put this in the task > https://phabricator.wikimedia.org/T101069#1379462 > > We'll probably want an instance per extension, to simplify having to > worry about dependencies (we can just run a git update on all > extensions after each checkout) > > On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez > <jhernan...@wikimedia.org> wrote: > > Awesome Jon! I'm so happy to finally see this developing :DD > > Loving the : `Browserbot happy!` > > I've noticed it can report either the name of the failing test or the > full > log. What do you think if we show that, and a url with the pasted log > somewhere publicly to not put too much noise on the comments but still > be > able to see it? Something like > https://phabricator.wikimedia.org/paste/... > > +1 to where is the source. > +1 to documenting how you've set it all up on wiki somewhere. > > I also think we need a catchy phrase for the -1s! > > Thanks for you work on this, we'll get more focused time for it soon. > > On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith <samsm...@wikimedia.org> > wrote: > > > I agree with Florian everything that you've written should be in a > public > version control system. > > Second, I'd ask that you document your experiences so far in getting > this > set up and how it works so that other members of the vertical can help > to > maintain it moving forward. > > Third, great work!!1 > > <3 > > –Sam > > On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.wel...@t-online.de > <florian.schmidt.wel...@t-online.de> wrote: > > > > It's currently working via a script that you can find here: > /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh > > > It would be great to have the script in a public version control > system > (e.g. github?), especially for people, e.g. volunteers, who can't ssh > to > gather-browser-tests.eqiad.wmflabs[1] > > [1] all people, who're not members of > https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests > > Best, > Florian > > -----Original-Nachricht----- > Betreff: [WikimediaMobile] [Update] Browser tests per patch > Datum: Thu, 18 Jun 2015 03:27:32 +0200 > Von: Jon Robson <jrob...@wikimedia.org> > An: "QA (software quality assurance) for Wikimedia projects." > <q...@lists.wikimedia.org>, mobile-l <mobile-l@lists.wikimedia.org> > > Background: mobile wants to gain more confidence in its browser tests > by running a subset of browser tests on a case by case basis [0]. > > Good news: I've got a proof of concept running and Barry the browser > test bot has given some legitimate helpful reviews to Gather [1]. > > Even better news: It's proving itself valuable already [2]. > As you can see in the messages the bot has posted on [3] we have a > couple of options on display option format for his reviews. > > So.. hopefully this short experience has sold you all already. > > This script is currently a manual job and needs a bit of tweaking > before we can put it in a cron job/run it always - it needs to watch > for new commits and then run a modification of the above script on a > per case basis (if two versions of it run in parallel we have an > issue). > > Definitely something we should push for next sprint! > > Long live Barry bot! > > Devs... (everyone else now of what follows is likely to be useful): > I got the labs instance up and running on: > http://gather-browser-tests.wmflabs.org/wiki/Main_Page > > Most of you in readership team should be able to ssh > gather-browser-tests.eqiad.wmflabs > Let me know if you have no access. > > It's currently working via a script that you can find here: > /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh > > [0] https://phabricator.wikimedia.org/T100293 > [1] > > > https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.com+status:open,n,z > [2] https://gerrit.wikimedia.org/r/#/c/218731/ > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > > > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > > -- > Jon Robson > * http://jonrobson.me.uk > * https://www.facebook.com/jonrobson > * @rakugojon > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > > > -- > Dan Duvall > Automation Engineer > Wikimedia Foundation > > > > > -- > Jon Robson > * http://jonrobson.me.uk > * https://www.facebook.com/jonrobson > * @rakugojon > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > >
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
