All - thanks for your detailed thoughts on this so far. It seems like this should've come with a fair bit of notice. I will hold on this separation for now. May be a couple more releases to see the level of participation in the QA and if it still feels like a hindrance we'll discuss separating it later. For now I think it should suffice to reorganize the current folder structure of marvin to be able to find all its bits at a single place. This will be proposed separately.
I am closing this DISCUSS thread as of now by not splitting out marvin. I will however address some additional points which were raised so far. Inline. On Fri, Oct 04, 2013 at 04:52:45PM -0400, Chip Childers wrote: > > Fair warning - some of this is a straw man argument to explore the > situation, and a little bit of ranting at the end. > > On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote: > > I'll summarize and address the concerns raised so far. > > > > Marvin has been in this repo for a long time for us to start writing > > tests. The only tests I've seen coming are from a specific set of > > people focussed on QA efforts. > > I agree - and that's a problem. New features should *ALL* have tests > before they merge into master. I think that assuming that the only test > writers are a group of folks that write the tests today is actually a > larger problem. > It's a bit more prevalent than test cases alone. Docs for eg are written by doc writers and there's a whole bit of process of doc review and someone writing the stuff in an email and then someone else going and correcting it. If you have repo access and notice a problem, please help fix it as well and not create additional process. I'm not for such segmentation but there are roles in a corporate setup that are probably causing this presumption that marvin tests are written by QA alone. > > I want to reduce the impediment for > > people who are writing tests *today*. Those looking to get started in > > the near future won't have any new learning to do, just that their > > code goes in an alternate repo that is pointed to right > > infrastructure. > > > > Automated testing also works in a push-to-production style very often. > > Testers need to run their tests on a deployed environment(s) quickly > > to be able to ensure it is valid and passes. By making them go through > > reviewboard each time for each test we massively slow down the > > process. (tons of fixes to tests are on rb today, not just new tests). > > We don't know if they run until they run on the environment. > > I want to be clear about this part - a different repo doesn't change the > need for someone to be a committer to commit. Yes - I thought this would be a problem as David also mentioned earlier. There are no ACLs within a project for controlling repo access. Except if it was possible, I would've looked at providing access faster to those who are contributing. > > > > Reason for tests and framework to go together is simple. If I go look > > at the jclouds repository today I find tests for rackspace cloud, > > openstack cloud, cloudstack cloud, euca clouds in the jclouds > > repository and not in the respective provider/project repository. A > > newcomer to the marvin repository will be someone interested in > > writing tests and he will also thus be able to find tests in the > > marvin repository. > > > > This also allows for more heterogenous testing of cloudstack. No one > > needs to be tied down to a framework / tool to write integration > > tests. If python is not your forte, use Chip's ruby client, or perhaps > > in the near future Chiradeep's stackmate to write your test, or even > > jclouds. > > But that's actually true today, right? I mean if I wanted to write an > integration test using some other method, I'd do that... but would it > be useful for others? Probably not! That's because the way that we do > testing of this type is via Marvin. The Citrix infra wouldn't be setup > for whatever other framework I used, and the community as a whole would > get less benefit than if I was consistent. > Somehow I sense that as a problem of our test writing. The tests are written to assume a certain infrastructure. But a lot of the API is also admin only which requires certain infra to be in place. The other tools jclouds live tests for example cannot assume admin access to the cloud because they are only testing the user api. Tests written by other tools would only be provided an endpoint and API/Secret keys and test whatever they can. Specific tools required for those tests can be provisioned automatically on the machine that runs the tests. > > > > Now the question of supporting older version of marvin against newer > > versions of cloudstack. Marvin now fully auto-generates itself (see > > the design in the proposal) based on endpoint. So you have the > > marvin version that will work with your endpoint only. As for being > > backwards compatible (also addressed in the design doc) - no old tests > > are broken, they will still run perfectly fine. > > > > That's marvin, not the tests, right? If I add a new feature for 4.3, > should that test appear in a 4.2 run? Aren't we aiming for (I know > we're not there) a situation where *all* tests pass before we release? > IMO not versioning the tests (not the Marvin framework) with the target > of the tests is confusing. We would mirror the branches of ACS release with the marvin repo to control what gets tested against what. Also, If an API is not available on a certain version of CloudStack and the test requires it, we can have the framework skip such a test. > > > The infrastructure (currently) only looks at the changes in the test > > directory before performing a run. It doesn't care whether server/ was > > changed or plugins/x/y/z was changed. That's because the tests are > > unrelated to what is in the rest of the repository. In fact you can't > > even run them without a deployed cloud. So I don't see why idle code > > should lie in the repo. > > > > Integration tests are essential, they will keep coming as long as > > Citrix QA is invested in the effort, but they need to come faster into > > the repos and that will be addressed by the separation IMO. > > How? Rewarding committership quickly. non sequitir given we cannot control who gets committership on which repo within a single project. > > > > What I would like to see provided by CloudStack is the ability to > > upgrade all our test environments, staging environments, UATs, what > > have you in continuous integration and have tests run on the > > upgraded setup. That allows incrementally testing CloudStack the way > > users do it. The current design of installing everything from scratch, > > redoing the testbed for each automated test run is mostly a workaround > > for that inability. If we had this ability tests written in marvin can > > be run against live setups at all times as and when features merge to > > master. > > This is a valid argument worth exploring. However, it doesn't require a > separate repo, right? I mean, the repo could be cloned twice if > necessary. > There is a different sense I'm getting off late where large refactors are done without enough unit-tests and we bank on our bvt to allow for a merge. The project requires both unittests and integration tests IMO. -- Prasanna., ------------------------ Powered by BigRock.com
