Fair warning - some of this is a straw man argument to explore the situation, and a little bit of ranting at the end.
On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote: > I'll summarize and address the concerns raised so far. > > Marvin has been in this repo for a long time for us to start writing > tests. The only tests I've seen coming are from a specific set of > people focussed on QA efforts. I agree - and that's a problem. New features should *ALL* have tests before they merge into master. I think that assuming that the only test writers are a group of folks that write the tests today is actually a larger problem. > I want to reduce the impediment for > people who are writing tests *today*. Those looking to get started in > the near future won't have any new learning to do, just that their > code goes in an alternate repo that is pointed to right > infrastructure. > > Automated testing also works in a push-to-production style very often. > Testers need to run their tests on a deployed environment(s) quickly > to be able to ensure it is valid and passes. By making them go through > reviewboard each time for each test we massively slow down the > process. (tons of fixes to tests are on rb today, not just new tests). > We don't know if they run until they run on the environment. I want to be clear about this part - a different repo doesn't change the need for someone to be a committer to commit. There's a thread on private@ that you should weigh in on here. If we have more people that should be committers, then let's get *that* done. > > Reason for tests and framework to go together is simple. If I go look > at the jclouds repository today I find tests for rackspace cloud, > openstack cloud, cloudstack cloud, euca clouds in the jclouds > repository and not in the respective provider/project repository. A > newcomer to the marvin repository will be someone interested in > writing tests and he will also thus be able to find tests in the > marvin repository. > > This also allows for more heterogenous testing of cloudstack. No one > needs to be tied down to a framework / tool to write integration > tests. If python is not your forte, use Chip's ruby client, or perhaps > in the near future Chiradeep's stackmate to write your test, or even > jclouds. But that's actually true today, right? I mean if I wanted to write an integration test using some other method, I'd do that... but would it be useful for others? Probably not! That's because the way that we do testing of this type is via Marvin. The Citrix infra wouldn't be setup for whatever other framework I used, and the community as a whole would get less benefit than if I was consistent. > > Now the question of supporting older version of marvin against newer > versions of cloudstack. Marvin now fully auto-generates itself (see > the design in the proposal) based on endpoint. So you have the > marvin version that will work with your endpoint only. As for being > backwards compatible (also addressed in the design doc) - no old tests > are broken, they will still run perfectly fine. > That's marvin, not the tests, right? If I add a new feature for 4.3, should that test appear in a 4.2 run? Aren't we aiming for (I know we're not there) a situation where *all* tests pass before we release? IMO not versioning the tests (not the Marvin framework) with the target of the tests is confusing. > The infrastructure (currently) only looks at the changes in the test > directory before performing a run. It doesn't care whether server/ was > changed or plugins/x/y/z was changed. That's because the tests are > unrelated to what is in the rest of the repository. In fact you can't > even run them without a deployed cloud. So I don't see why idle code > should lie in the repo. > > Integration tests are essential, they will keep coming as long as > Citrix QA is invested in the effort, but they need to come faster into > the repos and that will be addressed by the separation IMO. How? > Managing > the feature submitted to cloudstack against tests submitted to marvin > is not a hard thing to do. We simply mirror the release branches in > marvin and submit tests there. In fact I wonder why we didn't have > this question when docs were separated? It doesn't work any > differently really. Well, it turns out that docs are absolutely not being done before code comes into master. I dislike this fact, but live with it. > > What I would like to see provided by CloudStack is the ability to > upgrade all our test environments, staging environments, UATs, what > have you in continuous integration and have tests run on the > upgraded setup. That allows incrementally testing CloudStack the way > users do it. The current design of installing everything from scratch, > redoing the testbed for each automated test run is mostly a workaround > for that inability. If we had this ability tests written in marvin can > be run against live setups at all times as and when features merge to > master. This is a valid argument worth exploring. However, it doesn't require a separate repo, right? I mean, the repo could be cloned twice if necessary. > > It has always been my goal to reduce the barrier to writing tests for > cloudstack and that's entirely the goal of the refactor as well. So I > hope I have spent my time well. Since I can't be after people to write > tests, and setup mandates, I aim to address the technical difficulty > in writing tests through this refactor and separation. Some of my arguments above are, frankly, straw man arguments. I see many of your points as being valid as well. Don't take my perspective as being against any of your goals, just as discussion points about the right way to make the goals work. I'll say one thing though... I'm exceptionally frustrated with the features / commits coming into master with little to no tests (unit OR integration). I'm at a loss at this point, and frankly considering a possible reality that this community (generally) just doesn't care about making it easy to reproduce tests. The lack of interest and ignorance about the pain that this causes is infuriating to me. Prasanna, I can tell that you have some / similar frustrations (but I won't put the words into your mouth), and are trying to find ways to make it easier for others to step in and help out where feature developers are apparently showing no interest. This is an admirable goal. So... perhaps we have to live with the complete lack of interest from the feature devs, and take the approach of breaking out the code. That doesn't make *access permission* easier, but it does make a target repo easier to focus on test scenarios. As I said above, let's talk on private@ about people that are doing the tests that deserve the commit bit. At least we can help them help everyone else... > > PS: David, by hosted docs I mean something like > http://pythonhosted.org/cloudmonkey. Lives in the repo but read from > the webpage. > > -- > Prasanna., > > ------------------------ > Powered by BigRock.com > >
