Prasanna, I share Chips concern. But let me start by saying that given the effort Citrix puts in the integration testing, making it easy for your guys should be somewhat a priority. However, can we do the separation is such a way that integration tests as part of plugin and other enhancements is still trivial? for instance make the maven scripts checkout a stable marvin version and run tests using this framework. What Chip wants, i think, is that we enforce a integration test py in every contribution not only to validate the feature but also to ensure its continued functioning while the contributors may have wondered off in their $dayjob.
I followed the reviews submitted and know that the ones in the marvin corner of the world are a big chunk of it ans have few reviewers/committers in comparison. Can your refactor be organized so that integration tests can go in the marvin and in the cloudstack repo? thanks, Daan On Fri, Oct 4, 2013 at 2:27 PM, Prasanna Santhanam <t...@apache.org> wrote: > I'll summarize and address the concerns raised so far. > > Marvin has been in this repo for a long time for us to start writing > tests. The only tests I've seen coming are from a specific set of > people focussed on QA efforts. I want to reduce the impediment for > people who are writing tests *today*. Those looking to get started in > the near future won't have any new learning to do, just that their > code goes in an alternate repo that is pointed to right > infrastructure. > > Automated testing also works in a push-to-production style very often. > Testers need to run their tests on a deployed environment(s) quickly > to be able to ensure it is valid and passes. By making them go through > reviewboard each time for each test we massively slow down the > process. (tons of fixes to tests are on rb today, not just new tests). > We don't know if they run until they run on the environment. > > Reason for tests and framework to go together is simple. If I go look > at the jclouds repository today I find tests for rackspace cloud, > openstack cloud, cloudstack cloud, euca clouds in the jclouds > repository and not in the respective provider/project repository. A > newcomer to the marvin repository will be someone interested in > writing tests and he will also thus be able to find tests in the > marvin repository. > > This also allows for more heterogenous testing of cloudstack. No one > needs to be tied down to a framework / tool to write integration > tests. If python is not your forte, use Chip's ruby client, or perhaps > in the near future Chiradeep's stackmate to write your test, or even > jclouds. > > Now the question of supporting older version of marvin against newer > versions of cloudstack. Marvin now fully auto-generates itself (see > the design in the proposal) based on endpoint. So you have the > marvin version that will work with your endpoint only. As for being > backwards compatible (also addressed in the design doc) - no old tests > are broken, they will still run perfectly fine. > > The infrastructure (currently) only looks at the changes in the test > directory before performing a run. It doesn't care whether server/ was > changed or plugins/x/y/z was changed. That's because the tests are > unrelated to what is in the rest of the repository. In fact you can't > even run them without a deployed cloud. So I don't see why idle code > should lie in the repo. > > Integration tests are essential, they will keep coming as long as > Citrix QA is invested in the effort, but they need to come faster into > the repos and that will be addressed by the separation IMO. Managing > the feature submitted to cloudstack against tests submitted to marvin > is not a hard thing to do. We simply mirror the release branches in > marvin and submit tests there. In fact I wonder why we didn't have > this question when docs were separated? It doesn't work any > differently really. > > What I would like to see provided by CloudStack is the ability to > upgrade all our test environments, staging environments, UATs, what > have you in continuous integration and have tests run on the > upgraded setup. That allows incrementally testing CloudStack the way > users do it. The current design of installing everything from scratch, > redoing the testbed for each automated test run is mostly a workaround > for that inability. If we had this ability tests written in marvin can > be run against live setups at all times as and when features merge to > master. > > It has always been my goal to reduce the barrier to writing tests for > cloudstack and that's entirely the goal of the refactor as well. So I > hope I have spent my time well. Since I can't be after people to write > tests, and setup mandates, I aim to address the technical difficulty > in writing tests through this refactor and separation. > > PS: David, by hosted docs I mean something like > http://pythonhosted.org/cloudmonkey. Lives in the repo but read from > the webpage. > > -- > Prasanna., > > ------------------------ > Powered by BigRock.com > >
