On Wed, May 7, 2014 at 2:03 AM, Alex Huang <alex.hu...@citrix.com> wrote:
> Hi All, > > This is something I brought up a long time ago but really didn't have the > resources to get it all up and running until now. Throughout the past > year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have > been chipping away at it. At this point, we finally pull together a > continuous integration setup that we can use to make sure that CloudStack > master and the currently release branch are always stable. This is getting > pretty close to be completed and we like to share it with the community in > hopes that we can reduce/eliminate that problems we've seen with our recent > releases. Currently, the physical hardware are hosted by Citrix but we'll > be more than willing to donate the work to infra when that's all settled. > > This does require effort from the community to make a change in their > development process. These steps are detailed at [1]. I like to get > feedback on what everyone think about this. > > What have we done: > - We replaced a large selection of the BVT tests to run with the > simulator instead of actual hardware. This shortens the duration of each > BVT run. Today, a BVT that runs tests for XenServer and KVM completes in > 30-40 minutes. > How much is running with Simulator instead of actual hardware? My issue with this is that you're testing against a flawless simulator in terms of testing, while with actual hardware you are bound to hit bugs/issues that might not be due to ACS code but ACS still has to handle it. As an example, could you run a test on the tags '4.4.0' and '4.3.0' and report the result? They both had fundamental flaws, where the one was practically useless for a week or so, and the other had major issues with KVM, and if the BVT doesn't encounter those because it's using the simulator I see it as a burden rather than a gift, since you're relying on a false result. -- Erik
