On automated QA front following is available: 1. Before pushing in a feature a dev can run simulator based tests that will basically test various functionality that does not depend on the type of Hypervisor. (https://cwiki.apache.org/confluence/display/CLOUDSTACK/Validating+check-ins+for+your+local+changes%2C+using+Simulator.) The test suite are located in testing/integration/smoke folder. The travis system runs most of the test in this folder.
2. Then there are tests that will require real hardware to run most of these are in testing/integration/component. Basically there are two kind of test cases not strictly classified as per above directory structure - Ones that have required_hardware set as “false” and “others” that have this as “true”. The one with required_hardware is false can run on simulator but for the others you need a real Hypervisor based environment. I have been able to run a lot of tests both with hardware and simulator. The problem I faced is scattered documentation. Missing description of a model deployment; say for a particular Hypervisor that will allow a dev to run the provisioning tests. In all there is a huge scope for improvement. -abhi > On 23-Apr-2015, at 1:02 am, Remi Bergsma <r...@remi.nl> wrote: > > Hi, > > The '2 week cycle' was intended as something to work towards, like a > mission. The nice thing is that it immediately shows that we cannot achieve > such a thing if we keep our current method of (semi)manual testing, as you > said. Totally agree. > > That's why we need to improve on our automated functional testing. And that > will of course take time and effort. As I don't currently have a clear > overview of what is already available, I'll try to get that first and work > from there. I spoke to several people recently and most seem to do testing > on their own way (with sometimes cool automatic stuff going on!). It'd be > interesting to bring that together and share. > I think improving this is important, so I'll try to spend as much time on > this as possible. > > However, the tread started with comments on 4.5. Let's try to get it stable > and deliver 4.5.1. What is still to be done here? Can we share the load > somehow to fasten it? > > Regards, > Remi > > 2015-04-22 20:13 GMT+02:00 Paul Angus <paul.an...@shapeblue.com>: > >> I fully support the idea of a stable master with an automated CD process >> to protect against regressions. >> >> ....However, we obviously don't currently have fantastic integration >> testing otherwise we wouldn't have relied on 'people' to pick up the >> release blockers recently. A 2 week release cycle IMHO is way too frequent >> to expect 'people' to be carrying out integration testing. >> >> Maybe 1 month is a workable compromise until the integration testing is of >> a coverage and standard that can give real confidence. >> >> >> >> I'm also going to compile a list of people who vote "+1 - it works on my >> laptop" and devise a Guinness related punishment for any of them that show >> up at the CloudStack day in Dublin. >> >> Regards >> >> Paul Angus >> Cloud Architect >> S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus >> paul.an...@shapeblue.com >> >> -----Original Message----- >> From: Remi Bergsma [mailto:r...@remi.nl] >> Sent: 22 April 2015 10:25 >> To: dev@cloudstack.apache.org >> Subject: Re: Next ACS release? >> >> I'd be happy to help here as well. Last week in Austin, we also discussed >> this topic a couple of times. I do agree shorter release cycles are better. >> >> In my opinion, the first thing to improve is the minor releases in both the >> 4.4 and 4.5 branches. If we speed those up to let's say once every 2-weeks >> we will be able to do the next minor release with less effort and users can >> choose to either wait to start using 4.5 or start now and upgrade when the >> next minor release is available. This would have helped in getting 4.5 out >> on time and quickly fixing issues after the initial release. Also, it will >> save time which we can invest in getting the next release out on time, etc. >> >> The common thing here is we need more automated testing, preferably >> functional testing in addition to unit testing. There might also be other >> steps that we do manually now that can be automated. I'll try to understand >> the current process first and then come up with a proposal to improve which >> we can discuss. >> >> Regards, >> Remi >> >> 2015-04-22 10:56 GMT+02:00 Erik Weber <terbol...@gmail.com>: >> >>> On Wed, Apr 22, 2015 at 10:46 AM, Daan Hoogland >>> <daan.hoogl...@gmail.com> >>> wrote: >>> >>>> On Wed, Apr 22, 2015 at 10:29 AM, Erik Weber <terbol...@gmail.com> >>> wrote: >>>>> On Wed, Apr 22, 2015 at 9:49 AM, Stephen Turner < >>>> stephen.tur...@citrix.com> >>>>> wrote: >>>>> >>>>>> I have to admit I'm a bit sceptical because when we did have a >>>> four-month >>>>>> release cycle, we never seemed to manage to meet it. Personally I >>> think >>>>>> six-monthly might be easier. >>>>>> >>>>>> Having said that, part of the problem was the long close-down >>>>>> period >>>> where >>>>>> we kept finding critical bugs, so more automated testing might >>>>>> help to shorten the cycle. >>>>>> >>>>>> >>>>> >>>>> - Shorter cycle means that it's less of a problem to miss a >>>>> deadline, meaning you can spend more time on QA. >>>>> - Longer cycle means that it could be critical to meet the >>>>> deadline, meaning you might end up doing less QA while stressing >>>>> to get your >>>> feature >>>>> in. >>>>> >>>>> My $.002 >>>> >>>> >>> Yes, Eric and the amount of qa needed is less as well. Bare with me >>>> for a little rant here. >>>> >>>> When less new features are introduced less new interdependencies are >>>> introduced. I could try and expand on this but people make phd on >>>> the subject so that would be presumptuous of me. >>>> >>>> un-educated guess is (m + n)! - m! >>>> where m is old features and n is new features >>>> >>>> example: >>>> 1 old feature + 1 new feature mean 1 check to see if they (still) >>>> work >>>> 1 old feature + 2 new features means 3 checks to see if they all >>>> work >>>> 2 + 2 = 6 of which only 1 is old and should be fine >>>> 2 + 3 = 10, see the n! progressing ;) >>>> >>>> >>>> this is an over simplification as the complexity of features comes >>>> in play as well. I make them out for being function points here. >>>> those are fables of course. >>>> >>>> thanks for baring that. >>>> >>>> >>> I'm all with you on this Daan. >>> Just for the record, my notion of QA was meant at the feature >>> developer, ie. that they could use more time on test coverage etc. >>> without having to worry that much about feature freeze dates. >>> >>> -- >>> Erik >>> >> Find out more about ShapeBlue and our range of CloudStack related services >> >> IaaS Cloud Design & Build< >> http://shapeblue.com/iaas-cloud-design-and-build//> >> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> >> CloudStack Software Engineering< >> http://shapeblue.com/cloudstack-software-engineering/> >> CloudStack Infrastructure Support< >> http://shapeblue.com/cloudstack-infrastructure-support/> >> CloudStack Bootcamp Training Courses< >> http://shapeblue.com/cloudstack-training/> >> >> This email and any attachments to it may be confidential and are intended >> solely for the use of the individual to whom it is addressed. Any views or >> opinions expressed are solely those of the author and do not necessarily >> represent those of Shape Blue Ltd or related companies. If you are not the >> intended recipient of this email, you must neither take any action based >> upon its contents, nor copy or show it to anyone. Please contact the sender >> if you believe you have received this email in error. Shape Blue Ltd is a >> company incorporated in England & Wales. ShapeBlue Services India LLP is a >> company incorporated in India and is operated under license from Shape Blue >> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil >> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is >> a company registered by The Republic of South Africa and is traded under >> license from Shape Blue Ltd. ShapeBlue is a registered trademark. >> Find out more about ShapeBlue and our range of CloudStack related services IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
