> -----Original Message----- > From: Chip Childers [mailto:chip.child...@sungard.com] > Sent: Friday, October 04, 2013 1:53 PM > To: dev@cloudstack.apache.org > Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack > > > Fair warning - some of this is a straw man argument to explore the > situation, and a little bit of ranting at the end. > > On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote: > > I'll summarize and address the concerns raised so far. > > > > Marvin has been in this repo for a long time for us to start writing > > tests. The only tests I've seen coming are from a specific set of > > people focussed on QA efforts. > > I agree - and that's a problem. New features should *ALL* have tests > before they merge into master. I think that assuming that the only test > writers are a group of folks that write the tests today is actually a > larger problem.
> > > I want to reduce the impediment for > > people who are writing tests *today*. Those looking to get started in > > the near future won't have any new learning to do, just that their > > code goes in an alternate repo that is pointed to right > > infrastructure. > > > > Automated testing also works in a push-to-production style very often. > > Testers need to run their tests on a deployed environment(s) quickly > > to be able to ensure it is valid and passes. By making them go through > > reviewboard each time for each test we massively slow down the > > process. (tons of fixes to tests are on rb today, not just new tests). > > We don't know if they run until they run on the environment. > > I want to be clear about this part - a different repo doesn't change the > need for someone to be a committer to commit. > [Animesh>] Test code contribution is similar to code contribution and the issue is not reviewboard but folks not responding to reviews. We need to address them as community. If we have to get new committers we should bring them forward. > There's a thread on private@ that you should weigh in on here. If we > have more people that should be committers, then let's get *that* done. > > > > > Reason for tests and framework to go together is simple. If I go look > > at the jclouds repository today I find tests for rackspace cloud, > > openstack cloud, cloudstack cloud, euca clouds in the jclouds > > repository and not in the respective provider/project repository. A > > newcomer to the marvin repository will be someone interested in > > writing tests and he will also thus be able to find tests in the > > marvin repository. > > > > This also allows for more heterogenous testing of cloudstack. No one > > needs to be tied down to a framework / tool to write integration > > tests. If python is not your forte, use Chip's ruby client, or perhaps > > in the near future Chiradeep's stackmate to write your test, or even > > jclouds. > > But that's actually true today, right? I mean if I wanted to write an > integration test using some other method, I'd do that... but would it > be useful for others? Probably not! That's because the way that we do > testing of this type is via Marvin. The Citrix infra wouldn't be setup > for whatever other framework I used, and the community as a whole would > get less benefit than if I was consistent. > [Animesh>] I agree having consistent tooling shortens the learning cycle for new contributors > > > > Now the question of supporting older version of marvin against newer > > versions of cloudstack. Marvin now fully auto-generates itself (see > > the design in the proposal) based on endpoint. So you have the marvin > > version that will work with your endpoint only. As for being backwards > > compatible (also addressed in the design doc) - no old tests are > > broken, they will still run perfectly fine. > > > > That's marvin, not the tests, right? If I add a new feature for 4.3, > should that test appear in a 4.2 run? Aren't we aiming for (I know > we're not there) a situation where *all* tests pass before we release? > IMO not versioning the tests (not the Marvin framework) with the target > of the tests is confusing. > [Animesh>] IMO "Things that change together, should live together", separate repo for tests does not sit well > > The infrastructure (currently) only looks at the changes in the test > > directory before performing a run. It doesn't care whether server/ was > > changed or plugins/x/y/z was changed. That's because the tests are > > unrelated to what is in the rest of the repository. In fact you can't > > even run them without a deployed cloud. So I don't see why idle code > > should lie in the repo. > > > > Integration tests are essential, they will keep coming as long as > > Citrix QA is invested in the effort, but they need to come faster into > > the repos and that will be addressed by the separation IMO. > > How? [Animesh>] Like Chip, I don't see breaking tests into separate repo will increase their velocity. > > > Managing > > the feature submitted to cloudstack against tests submitted to marvin > > is not a hard thing to do. We simply mirror the release branches in > > marvin and submit tests there. In fact I wonder why we didn't have > > this question when docs were separated? It doesn't work any > > differently really. > > Well, it turns out that docs are absolutely not being done before code > comes into master. I dislike this fact, but live with it. > > > > > What I would like to see provided by CloudStack is the ability to > > upgrade all our test environments, staging environments, UATs, what > > have you in continuous integration and have tests run on the upgraded > > setup. That allows incrementally testing CloudStack the way users do > > it. The current design of installing everything from scratch, redoing > > the testbed for each automated test run is mostly a workaround for > > that inability. If we had this ability tests written in marvin can be > > run against live setups at all times as and when features merge to > > master. > > This is a valid argument worth exploring. However, it doesn't require a > separate repo, right? I mean, the repo could be cloned twice if > necessary. > > > > > It has always been my goal to reduce the barrier to writing tests for > > cloudstack and that's entirely the goal of the refactor as well. So I > > hope I have spent my time well. Since I can't be after people to write > > tests, and setup mandates, I aim to address the technical difficulty > > in writing tests through this refactor and separation. > > Some of my arguments above are, frankly, straw man arguments. I see > many of your points as being valid as well. Don't take my perspective > as being against any of your goals, just as discussion points about the > right way to make the goals work. > > I'll say one thing though... I'm exceptionally frustrated with the > features / commits coming into master with little to no tests (unit OR > integration). I'm at a loss at this point, and frankly considering a > possible reality that this community (generally) just doesn't care about > making it easy to reproduce tests. The lack of interest and ignorance > about the pain that this causes is infuriating to me. > > Prasanna, I can tell that you have some / similar frustrations (but I > won't put the words into your mouth), and are trying to find ways to > make it easier for others to step in and help out where feature > developers are apparently showing no interest. This is an admirable > goal. > > So... perhaps we have to live with the complete lack of interest from > the feature devs, and take the approach of breaking out the code. That > doesn't make *access permission* easier, but it does make a target repo > easier to focus on test scenarios. > > As I said above, let's talk on private@ about people that are doing the > tests that deserve the commit bit. At least we can help them help > everyone else... > > > > > PS: David, by hosted docs I mean something like > > http://pythonhosted.org/cloudmonkey. Lives in the repo but read from > > the webpage. > > > > -- > > Prasanna., > > > > ------------------------ > > Powered by BigRock.com > > > >
