One thing I didn't mention is that we recently marked a few of our selenium tests as smoke tests which I just spotted was suggested in the other email thread for core!
Great idea! Addshore On 7 March 2014 19:34, addshorewiki <addshorew...@gmail.com> wrote: > I feel like I should probably post here about the current Wikibase / > Wikidata deployment pipeline too which probably differs slightly to other > products. > > On a per commit basis: > A commit is made, the unit tests run on jenkins, the commit is reviewed, > amended, merged, Jenkins again runs the unit tests as gate submit, Travis > also runs the unit tests post merge testing against php 5.3, 5.4, 5.5 and > both sqlite and mysql. > > Daily at 10AM UTC: > Our build process is triggered and creates a build for Wikidata, Both the > WMF jenkins and our WMDE jenkins run the unit tests, if both pass we +2CR > and it is merged, this is then deployed straight to beta where our Jenkins > runs all of our Selenium tests, the tests report to the build gerrit commit > depending on their outcome. > > Branch Day (even 2 weeks): > Tuesday is generally branch day, we branch the Wikidata repo all the unit > tests and selenium tests are re run, we deploy to test.wikidata, manually > test and on thursday and wikidata.org the following tuesday. > > We have very good test coverage which generally makes everything much > easier! I probably missed something of interest above but generally > everything is covered. > > Addshore > > > On 7 March 2014 19:08, Greg Grossmeier <g...@wikimedia.org> wrote: > >> <quote name="Jon Robson" date="2014-03-07" time="09:30:09 -0800"> >> > Let's also take this into a new thread. There are a lot of different >> > conversations now going on.... >> >> >> My opinion is that fixing this with policy is going to be hard. >> >> Either everyone who commits needs to be mindful of what day/time it is >> and whether or not another human has cut the new branch yet (which isn't >> set in stone on when, it varies by a couple hours, depending on a lot of >> factors), OR we modify the branch cut based on some arbitrary offset >> (24 hours ago) or some human looks at the merges and picks a point. >> >> None of those are ideal/scalable. >> >> What we should do, however, is have a true "deployment pipeline". >> Briefly defined: A deployment pipeline is a sequence of events that >> increase your confidence in the quality of any particular build/commit >> point. >> >> A typical example is: >> commit -> unit tests -> integration tests -> manual tests -> release >> >> >> Each step has the ability to fail a build, which means "You shall not >> pass!" to that commit point. The earlier you get a "You shall not pass!" >> the better because it means less time waiting by the developers to know >> if what they committed is ok or not. >> >> >> What this means for us: >> The Mobile team is actually a good example. They are doing The Right >> Thing and have a lot of tests written, including browser tests. They run >> into problems when, eg: they write a new feature and associated test and >> commit it. >> >> Beta Cluster gets that code (feature and test) within 5 minutes. >> >> But, test.wikipedia and en.wikipedia get that feature much later, days >> later. >> >> However, the test code is run by Jenkins across all environments (beta >> cluster, test.wikipedia, en.wikipedia etc) all the time. So, the mobile >> team gets a ton of false positives when their new test runs against eg >> production where the feature isn't enabled yet (on purpose). >> >> The QA team is working on this problem now (loosely termed the >> "versioned test problem"). >> >> >> How a pipeline would help: >> >> Really, a pipeline isn't a thing like your indoor plumping but more of a >> mindset/way of designing your test infrastructure. But, it means that >> you keep things self-contained (contrary to the mobile example above) >> and things progress through the pipeline in a predicable way/pace. >> >> It also means that each code commit spends the exact same amount of time >> in the various stages as other code commits. Right now some code sits on >> Beta Cluster for 7 days before hitting production, whereas other code >> spends 0-5 minutes. That's not good. >> >> Wanna help us on this problem? We're hiring: >> https://hire.jobvite.com/Jobvite/jobvite.aspx?b=nHZ0zmw6 >> (2 job openings) >> >> >> Greg >> >> >> -- >> | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | >> | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D | >> >> _______________________________________________ >> Wikitech-l mailing list >> Wikitech-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l >> > > _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
