Hi Paul, Indeed not all Marvin tests are generic enough and can be run on all environments. If you use this [1] collection of ~85 tests it should work well as we tested the PRs with these. Sure we should add more, but it’s probably way faster if you’d use these and work from there, instead of trying to get them all to work.
Regards, Remi [1] https://github.com/schubergphilis/MCT-shared/blob/master/helper_scripts/cloudstack/run_marvin_router_tests.sh (the name of the file by now is a bit mis-leading) From: Paul Angus <paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com>> Reply-To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> Date: Thursday 28 January 2016 09:29 To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> Subject: RE: [PROPOSAL] Freeze everything until we get CI Hi, I mentioned last week I am working with a few of my colleagues to develop a repo of the Ansible plays to be able deploy both Marvin and virtualised environments - (relatively) simply and then run Marvin against that environment. As I said, we have a load of work to do to get to the bottom of issues, as currently a large proportion of the 250 Marvin tests fail in my 'early' builds. Once we triage issues into Marvin build/Marvin Tests/CloudStack Env/CloudStack Code, we'll be in better shape to see where we are. Our intention is to not only share our approach and code, but also (with everyone's blessing) to work to integrate it into the infra that Citrix donated to Apache and allow anyone to ask this 'communal marvin' to generate a specified environment and run any of the integration tests against master or any given PR.... We are actively working on this, but it is going to take a little time.... [ShapeBlue]<http://www.shapeblue.com> Paul Angus VP Technology , ShapeBlue d: +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540> | m: +44 7711 418784<tel:+44%207711%20418784> e: paul.an...@shapeblue.com | t: @cloudyangus<mailto:paul.an...@shapeblue.com%20|%20t:%20@cloudyangus> | w: www.shapeblue.com<http://www.shapeblue.com> a: 53 Chandos Place, Covent Garden London WC2N 4HS UK [cid:imagec996fc.png@836cfe96.47bdedb6] Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. -----Original Message----- From: Bharat Kumar [mailto:bharat.ku...@citrix.com] Sent: 28 January 2016 14:45 To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> Subject: Re: [PROPOSAL] Freeze everything until we get CI Hi, I agree with Remi on the hurdles he mentioned. It is difficult to integrate 3rd party CI, If someone wants to implement their own CI, the link below gives one way to do it. https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration Thanks, Bharat. On 28-Jan-2016, at 2:52 AM, Wido den Hollander <w...@widodh.nl<mailto:w...@widodh.nl><mailto:w...@widodh.nl>> wrote: On 01/27/2016 10:02 PM, Remi Bergsma wrote: Hi all, We should keep the simple approach that was used until now: one LGTM based on code review and one LGTM based on integration tests (that’s not the same as 2xLGTM). If we care about master stability, every change has to be tested for regression. Period. Things may look OK, but still break something else in an unexpected way. IMHO, giving up on it would be a shame. In a perfect world, CI would automate this for us and run it against every PR. Current impediments I see for proper CI on every PR: - there is no community hardware to run integration tests on (there was supposed to be something) - no support from Apache Infra: need Github write access (also to hook 3rd party CI). It’s nowhere close. - the community has very few people that are able to run the integration tests on their own laptop/hardware and post the results on PRs We tried to address points 1 and 2 and I propose to give up on them. If all devs would stop developing for a couple of days and make sure they have a box that can run KVM, then clone the code we use [1] it should be pretty straight forward to run the integration tests. We choose to make it all virtual and used nested-virtualisation, but that is optional. A simple Intel Nuc or similar will do. Yes, that's great code. I still have to master it, but the README should get us there. I still need to create one which also spins up a Ceph cluster. Probably a good thing to do now ;) Wido People can then test the PRs that they find interesting and post results, after which it can be merged. Distribute it. Share the load. This is what Wilder, Miguel, Boris, Daan an I have been doing for months. Simply running the tests. Hundreds of times. That’s why we can run a 100% Mission Critical Cloud close to master branch at Schuberg Philis. Regards, Remi [1] https://github.com/schubergphilis/MCT-shared/ On 27/01/16 21:33, "Sebastien Goasguen" <run...@gmail.com<mailto:run...@gmail.com><mailto:run...@gmail.com>> wrote: On Jan 27, 2016, at 9:25 PM, Wido den Hollander <w...@widodh.nl<mailto:w...@widodh.nl><mailto:w...@widodh.nl>> wrote: On 01/27/2016 09:18 PM, Sebastien Goasguen wrote: Folks, How about we freeze our repo entirely until we get proper CI in place. Seems to me all the hard work from Remi and co could be lost if we start committing again. Now Travis is not running, Jenkins fails all the time and nobody cares… So how about we figure out CI now ? and not do anything else. I think forces have to be combined to make this work. Questions which come to mind: Who runs Jenkins? Do we need a additional slave? I haven't figured out the Integration tests completely personally. In an ideal case, PR should trigger tests totally distributed on everyone’s own hardware. Then tests would report back on the PR. Only when all are green can we merge. there is an issue with creating triggers in github on our own, but I think that’s what we should aspire to. for instance, how can pcextreme automate its testing and report back on each PR ? Wido -Sebastien Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
