I'll summarize and address the concerns raised so far. Marvin has been in this repo for a long time for us to start writing tests. The only tests I've seen coming are from a specific set of people focussed on QA efforts. I want to reduce the impediment for people who are writing tests *today*. Those looking to get started in the near future won't have any new learning to do, just that their code goes in an alternate repo that is pointed to right infrastructure.
Automated testing also works in a push-to-production style very often. Testers need to run their tests on a deployed environment(s) quickly to be able to ensure it is valid and passes. By making them go through reviewboard each time for each test we massively slow down the process. (tons of fixes to tests are on rb today, not just new tests). We don't know if they run until they run on the environment. Reason for tests and framework to go together is simple. If I go look at the jclouds repository today I find tests for rackspace cloud, openstack cloud, cloudstack cloud, euca clouds in the jclouds repository and not in the respective provider/project repository. A newcomer to the marvin repository will be someone interested in writing tests and he will also thus be able to find tests in the marvin repository. This also allows for more heterogenous testing of cloudstack. No one needs to be tied down to a framework / tool to write integration tests. If python is not your forte, use Chip's ruby client, or perhaps in the near future Chiradeep's stackmate to write your test, or even jclouds. Now the question of supporting older version of marvin against newer versions of cloudstack. Marvin now fully auto-generates itself (see the design in the proposal) based on endpoint. So you have the marvin version that will work with your endpoint only. As for being backwards compatible (also addressed in the design doc) - no old tests are broken, they will still run perfectly fine. The infrastructure (currently) only looks at the changes in the test directory before performing a run. It doesn't care whether server/ was changed or plugins/x/y/z was changed. That's because the tests are unrelated to what is in the rest of the repository. In fact you can't even run them without a deployed cloud. So I don't see why idle code should lie in the repo. Integration tests are essential, they will keep coming as long as Citrix QA is invested in the effort, but they need to come faster into the repos and that will be addressed by the separation IMO. Managing the feature submitted to cloudstack against tests submitted to marvin is not a hard thing to do. We simply mirror the release branches in marvin and submit tests there. In fact I wonder why we didn't have this question when docs were separated? It doesn't work any differently really. What I would like to see provided by CloudStack is the ability to upgrade all our test environments, staging environments, UATs, what have you in continuous integration and have tests run on the upgraded setup. That allows incrementally testing CloudStack the way users do it. The current design of installing everything from scratch, redoing the testbed for each automated test run is mostly a workaround for that inability. If we had this ability tests written in marvin can be run against live setups at all times as and when features merge to master. It has always been my goal to reduce the barrier to writing tests for cloudstack and that's entirely the goal of the refactor as well. So I hope I have spent my time well. Since I can't be after people to write tests, and setup mandates, I aim to address the technical difficulty in writing tests through this refactor and separation. PS: David, by hosted docs I mean something like http://pythonhosted.org/cloudmonkey. Lives in the repo but read from the webpage. -- Prasanna., ------------------------ Powered by BigRock.com
