That's gerrit part of the proposal. I don't have enough karma with Infra to make that work. Dave or someone else need to take that up.
--Alex > -----Original Message----- > From: Kelven Yang [mailto:kelven.y...@citrix.com] > Sent: Wednesday, March 6, 2013 10:22 AM > To: cloudstack-dev@incubator.apache.org > Subject: Re: [PROPOSAL] BVT for CloudStack checkins > > First +1 on BVT. > > Second, should we consider the idea of having a staging area for people to > check-in? Which is that making master always the stable(reasonable) branch > for main development, but whenever people make check-ins, it goes into > staging first, and we have maintainers(could be automatic) to run whatever > test framework we have and only perform automatic merge to master from > staging area after a successful test-run? > > Kelven > > On 3/6/13 4:21 AM, "Prasanna Santhanam" <t...@apache.org> wrote: > > >Great to see you kick this off Alex! I have a few ideas for this and > >had been looking for help as well .. > > > >I had a draft lying around of an email you sent me long ago about > >tiered testing and I think your proposal fits very well on this: > > > >The tl;dr of that conversation was as below > > > >- improve simulator for runtime testability > >- customize to model and inject failures > >- make a habit of writing tests around bug reports (I started tagging > >tests since api_refactoring on JIRA already. > >look for the integration-test label) > >- make integration testing easier using factories and DSLs (from > > Chiradeep) > >(part 1 of this work was started on this in the marvin-refactor > >branch) > > > >More comments inline > > > >On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote: > >> Hi All, > >> > >> As most of you are aware, the master branch keeps getting broken by > >> checkins for various reasons. Committers need to be more responsible > >> about their checkins but I don't think we can depend on that > >> happening. There are various reasons. The most obvious to me is > >> that granting committership is not based on code competency. > >> (And I don't think it should.) Given that we need to build a BVT > >> system for ensure that checkins do not break the branch. > >> > >> Here's my proposal: > >> > >> Existing components that we'll use. > >> > >> - Citrix has contributed its testing to Apache. > >> > >> - Apache CloudStack has already a simulator that's been used > >>for scale testing. > >> > >> - Marvin > >> > >> - DevCloud-kvm > >> > >> Work Proposal: > >> > >> - Convert the Citrix testing into three phases: > >> > >> o Setup > >> > >> o Test > >> > >> o Verify > > > >I do the build, package, setup, test, verify in my integration test > >pipeline and a similar pipeline for a developer combined into easily > >runnable maven profiles/lifecycles/goals would be great to have. > > > >> - Add a Setup and Verify phase for the simulator > >> - Add all of the agent commands necessary for the simulator to > >>pass the testing. > >> - Add a Setup and Verify phase for devCloud-kvm > > > >The setups exists as configs in the marvin sandbox already. Just need > >to hook it up with mvn. For verify there is a simple python script > >testSetupSuccess.py already that checks two things > > > >- system VMs are up > >- built-in templates are downloaded > > > >This should be a good start IMO. > > > >Currently devcloud-kvm is a bit hard to run from a developer > >environment. But it's great to have in a continuous environment backed > >by a KVM host with a Linux 3.0 kernel. Marcus has written up some > >pretty good documentation for this. If someone can help bring up that > >setup I can assist in bringing up the tests using my devcloud-ci > >scripts. I'm bringing up devcloud after Kelven 'alerted' us of the > >memory fix. > > > >> > >> - Add two more profiles to pom > >> o Checkin-test-with-simulator: Runs the testing against the simulator > >> o Checkin-test-with-devCloud: Runs the testing against devcloud > >> > >> - All of the profiles will attempt to also check the merge > >>list that Chip has proposed.' > > > >> - We will also change marvin to easily add zones with > >> actual hardware. It will be based on a data driven document to do > >> the setup. > >This is currently partly doable using a properties file in the sandbox > >$ python advanced_env.py -i xen.properties -o xen.cfg > > > >gives out a advanced zone config for the properties specified. > >Is data driven similar or you are talking about a DSL that Edison > >mentioned at CCC12? > > > >> > >> For a developer to checkin: > >> > >> - S/he must writes marvin tests for their feature and add > >> it to the BVT. > >I made some progress on this in my marvin-refactor branch and will > >collect my thoughts in an FS that I am drafting. The marvin tests are > >difficult to write in their current form. I'm following Chiradeep's > >recommendation for providing factories and once those are baked a DSL > >language devs can quickly mock down tests against a simulator using the > >above mentioned mvn profiles. > > > >> > >> - S/he must run the checkin tests to verify everything works. > >> - If the feature contains a hardware/vpx component, simulation > >>must be added. > >> > >> At this point, everything is about the developer writing in their > >>feature branch and merging in. > >> > >> On infrastructure side: > >> > >> - We'll setup continuous BVT based on the simulator. > >I brought this up on the IRC and the list y'day, so +1 - happy to help > >> > >> - I again push that we must use Gerrit to test the code > >> before it gets merge into the branch but I'll leave that for someone > >> else to do that. > >> > >> Let me know what you guys think. I'll probably break out a bvt > >> branch to work on this. Anyone want to join me? > > > >Count me in! > > > >-- > >Prasanna.,
