@bhinandan and others, note that we have http://jenkins.buildacloud.org/view/parameterized/job/parameterized-slowbuild/ and https://builds.apache.org/view/A-D/view/Cloudstack/job/cloudstack-pull-requests/
those should give people a good idea on whether a branch can be merged to master based on simulator. Of course this is only a first step but let's use them intensively. On Thu, Apr 23, 2015 at 11:56 AM, Ian Southam <isout...@schubergphilis.com> wrote: > At SBP, we are working on two environments for testing. Primarily focussed > on Xen and KVM with Nicira and without Nicira. > > One will focus more on building and extending the current integration tests > and making sure they are running on real hardware in real life situations. > The other we are planning to be more “chaos monkey” in operation. So > genuinely testing how the system reacts to hypervisors crashing. Loss of > network connectivity, VR failures/failovers and so on. > > I know other people in the community are doing similar things. By having > enough such systems that together have a high coverage of all the various > configuration and hardware combinations out there, we should be able to > really get to a much shorter delivery cycle with quite high levels of > confidence about quality. > > We are not there yet but, I am absolutely convinced that aiming for a 2 week > release cycle is the way to go. > > — > Grts! > Ian > >> On 23 Apr 2015, at 10:38, Abhinandan Prateek >> <abhinandan.prat...@shapeblue.com> wrote: >> >> On automated QA front following is available: >> >> 1. Before pushing in a feature a dev can run simulator based tests that will >> basically test various functionality that does not depend on the type of >> Hypervisor. >> (https://cwiki.apache.org/confluence/display/CLOUDSTACK/Validating+check-ins+for+your+local+changes%2C+using+Simulator.) >> The test suite are located in testing/integration/smoke folder. >> The travis system runs most of the test in this folder. >> >> >> 2. Then there are tests that will require real hardware to run most of these >> are in testing/integration/component. >> >> >> Basically there are two kind of test cases not strictly classified as per >> above directory structure - Ones that have required_hardware set as “false” >> and “others” that have this as “true”. >> The one with required_hardware is false can run on simulator but for the >> others you need a real Hypervisor based environment. >> >> I have been able to run a lot of tests both with hardware and simulator. The >> problem I faced is scattered documentation. Missing description of a model >> deployment; say for a particular Hypervisor that will allow a dev to run the >> provisioning tests. >> >> In all there is a huge scope for improvement. >> >> >> -abhi >> >> >>> On 23-Apr-2015, at 1:02 am, Remi Bergsma <r...@remi.nl> wrote: >>> >>> Hi, >>> >>> The '2 week cycle' was intended as something to work towards, like a >>> mission. The nice thing is that it immediately shows that we cannot achieve >>> such a thing if we keep our current method of (semi)manual testing, as you >>> said. Totally agree. >>> >>> That's why we need to improve on our automated functional testing. And that >>> will of course take time and effort. As I don't currently have a clear >>> overview of what is already available, I'll try to get that first and work >>> from there. I spoke to several people recently and most seem to do testing >>> on their own way (with sometimes cool automatic stuff going on!). It'd be >>> interesting to bring that together and share. >>> I think improving this is important, so I'll try to spend as much time on >>> this as possible. >>> >>> However, the tread started with comments on 4.5. Let's try to get it stable >>> and deliver 4.5.1. What is still to be done here? Can we share the load >>> somehow to fasten it? >>> >>> Regards, >>> Remi >>> >>> 2015-04-22 20:13 GMT+02:00 Paul Angus <paul.an...@shapeblue.com>: >>> >>>> I fully support the idea of a stable master with an automated CD process >>>> to protect against regressions. >>>> >>>> ....However, we obviously don't currently have fantastic integration >>>> testing otherwise we wouldn't have relied on 'people' to pick up the >>>> release blockers recently. A 2 week release cycle IMHO is way too frequent >>>> to expect 'people' to be carrying out integration testing. >>>> >>>> Maybe 1 month is a workable compromise until the integration testing is of >>>> a coverage and standard that can give real confidence. >>>> >>>> >>>> >>>> I'm also going to compile a list of people who vote "+1 - it works on my >>>> laptop" and devise a Guinness related punishment for any of them that show >>>> up at the CloudStack day in Dublin. >>>> >>>> Regards >>>> >>>> Paul Angus >>>> Cloud Architect >>>> S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus >>>> paul.an...@shapeblue.com >>>> >>>> -----Original Message----- >>>> From: Remi Bergsma [mailto:r...@remi.nl] >>>> Sent: 22 April 2015 10:25 >>>> To: dev@cloudstack.apache.org >>>> Subject: Re: Next ACS release? >>>> >>>> I'd be happy to help here as well. Last week in Austin, we also discussed >>>> this topic a couple of times. I do agree shorter release cycles are better. >>>> >>>> In my opinion, the first thing to improve is the minor releases in both the >>>> 4.4 and 4.5 branches. If we speed those up to let's say once every 2-weeks >>>> we will be able to do the next minor release with less effort and users can >>>> choose to either wait to start using 4.5 or start now and upgrade when the >>>> next minor release is available. This would have helped in getting 4.5 out >>>> on time and quickly fixing issues after the initial release. Also, it will >>>> save time which we can invest in getting the next release out on time, etc. >>>> >>>> The common thing here is we need more automated testing, preferably >>>> functional testing in addition to unit testing. There might also be other >>>> steps that we do manually now that can be automated. I'll try to understand >>>> the current process first and then come up with a proposal to improve which >>>> we can discuss. >>>> >>>> Regards, >>>> Remi >>>> >>>> 2015-04-22 10:56 GMT+02:00 Erik Weber <terbol...@gmail.com>: >>>> >>>>> On Wed, Apr 22, 2015 at 10:46 AM, Daan Hoogland >>>>> <daan.hoogl...@gmail.com> >>>>> wrote: >>>>> >>>>>> On Wed, Apr 22, 2015 at 10:29 AM, Erik Weber <terbol...@gmail.com> >>>>> wrote: >>>>>>> On Wed, Apr 22, 2015 at 9:49 AM, Stephen Turner < >>>>>> stephen.tur...@citrix.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I have to admit I'm a bit sceptical because when we did have a >>>>>> four-month >>>>>>>> release cycle, we never seemed to manage to meet it. Personally I >>>>> think >>>>>>>> six-monthly might be easier. >>>>>>>> >>>>>>>> Having said that, part of the problem was the long close-down >>>>>>>> period >>>>>> where >>>>>>>> we kept finding critical bugs, so more automated testing might >>>>>>>> help to shorten the cycle. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> - Shorter cycle means that it's less of a problem to miss a >>>>>>> deadline, meaning you can spend more time on QA. >>>>>>> - Longer cycle means that it could be critical to meet the >>>>>>> deadline, meaning you might end up doing less QA while stressing >>>>>>> to get your >>>>>> feature >>>>>>> in. >>>>>>> >>>>>>> My $.002 >>>>>> >>>>>> >>>>> Yes, Eric and the amount of qa needed is less as well. Bare with me >>>>>> for a little rant here. >>>>>> >>>>>> When less new features are introduced less new interdependencies are >>>>>> introduced. I could try and expand on this but people make phd on >>>>>> the subject so that would be presumptuous of me. >>>>>> >>>>>> un-educated guess is (m + n)! - m! >>>>>> where m is old features and n is new features >>>>>> >>>>>> example: >>>>>> 1 old feature + 1 new feature mean 1 check to see if they (still) >>>>>> work >>>>>> 1 old feature + 2 new features means 3 checks to see if they all >>>>>> work >>>>>> 2 + 2 = 6 of which only 1 is old and should be fine >>>>>> 2 + 3 = 10, see the n! progressing ;) >>>>>> >>>>>> >>>>>> this is an over simplification as the complexity of features comes >>>>>> in play as well. I make them out for being function points here. >>>>>> those are fables of course. >>>>>> >>>>>> thanks for baring that. >>>>>> >>>>>> >>>>> I'm all with you on this Daan. >>>>> Just for the record, my notion of QA was meant at the feature >>>>> developer, ie. that they could use more time on test coverage etc. >>>>> without having to worry that much about feature freeze dates. >>>>> >>>>> -- >>>>> Erik >>>>> >>>> Find out more about ShapeBlue and our range of CloudStack related services >>>> >>>> IaaS Cloud Design & Build< >>>> http://shapeblue.com/iaas-cloud-design-and-build//> >>>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> >>>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> >>>> CloudStack Software Engineering< >>>> http://shapeblue.com/cloudstack-software-engineering/> >>>> CloudStack Infrastructure Support< >>>> http://shapeblue.com/cloudstack-infrastructure-support/> >>>> CloudStack Bootcamp Training Courses< >>>> http://shapeblue.com/cloudstack-training/> >>>> >>>> This email and any attachments to it may be confidential and are intended >>>> solely for the use of the individual to whom it is addressed. Any views or >>>> opinions expressed are solely those of the author and do not necessarily >>>> represent those of Shape Blue Ltd or related companies. If you are not the >>>> intended recipient of this email, you must neither take any action based >>>> upon its contents, nor copy or show it to anyone. Please contact the sender >>>> if you believe you have received this email in error. Shape Blue Ltd is a >>>> company incorporated in England & Wales. ShapeBlue Services India LLP is a >>>> company incorporated in India and is operated under license from Shape Blue >>>> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil >>>> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is >>>> a company registered by The Republic of South Africa and is traded under >>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark. >>>> >> >> Find out more about ShapeBlue and our range of CloudStack related services >> >> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> >> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> >> CloudStack Software >> Engineering<http://shapeblue.com/cloudstack-software-engineering/> >> CloudStack Infrastructure >> Support<http://shapeblue.com/cloudstack-infrastructure-support/> >> CloudStack Bootcamp Training >> Courses<http://shapeblue.com/cloudstack-training/> >> >> This email and any attachments to it may be confidential and are intended >> solely for the use of the individual to whom it is addressed. Any views or >> opinions expressed are solely those of the author and do not necessarily >> represent those of Shape Blue Ltd or related companies. If you are not the >> intended recipient of this email, you must neither take any action based >> upon its contents, nor copy or show it to anyone. Please contact the sender >> if you believe you have received this email in error. Shape Blue Ltd is a >> company incorporated in England & Wales. ShapeBlue Services India LLP is a >> company incorporated in India and is operated under license from Shape Blue >> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil >> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a >> company registered by The Republic of South Africa and is traded under >> license from Shape Blue Ltd. ShapeBlue is a registered trademark. > -- Daan
