H Chip, As a feature-dev driven by a 150-man-strong-cloud-operator-base that is not interested in anything but me showing that they can work with what they asked for:
On Fri, Oct 4, 2013 at 10:52 PM, Chip Childers <chip.child...@sungard.com>wrote: ... > On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote: > > I'll summarize and address the concerns raised so far. > > > > Marvin has been in this repo for a long time for us to start writing > > tests. The only tests I've seen coming are from a specific set of > > people focussed on QA efforts. > > I agree - and that's a problem. New features should *ALL* have tests > before they merge into master. I think that assuming that the only test > writers are a group of folks that write the tests today is actually a > larger problem. > Yes and we will need to work down a backlog of scenarios before we ever can rely on guys like me doing that. Not because they won't but because there is to much to write tests for edging on the new features they write. Just because those tests aren't there yet. I think giving Citrix QA a repo to work on is fine but I would like to see it merged back at some point and a continued possibility to write them in the main tree. > > > I want to reduce the impediment for > > people who are writing tests *today*. Those looking to get started in > > the near future won't have any new learning to do, just that their > > code goes in an alternate repo that is pointed to right > > infrastructure. > @prasanna: education of the developers is also something to consider. I know I could contribute more to integration and at Schuberg Philis we have some work going on in that direction. It would be nice to see easy integration of this with the Citrix work. > ... > > > > Reason for tests and framework to go together is simple. > Guys, why not both. some general tests that will be valid for any ACS version can go in the marvin code base. Others that change over time should be tied to the version of ACS they apply to. Test can be moved from one repo to the other if needed. ... > > That's marvin, not the tests, right? If I add a new feature for 4.3, > should that test appear in a 4.2 run? Aren't we aiming for (I know > we're not there) a situation where *all* tests pass before we release? > IMO not versioning the tests (not the Marvin framework) with the target > of the tests is confusing. > That I agree with completely. > > The infrastructure (currently) only looks at the changes in the test > > directory before performing a run. It doesn't care whether server/ was > > changed or plugins/x/y/z was changed. That's because the tests are > > unrelated to what is in the rest of the repository. In fact you can't > > even run them without a deployed cloud. So I don't see why idle code > > should lie in the repo. > You can trigger the building of a cloud when code changes and run the tests against that. > > > > Integration tests are essential, they will keep coming as long as > > Citrix QA is invested in the effort, but they need to come faster into > > the repos and that will be addressed by the separation IMO. > > How? > > > Managing > > the feature submitted to cloudstack against tests submitted to marvin > > is not a hard thing to do. We simply mirror the release branches in > > marvin and submit tests there. In fact I wonder why we didn't have > > this question when docs were separated? It doesn't work any > > differently really. > > Well, it turns out that docs are absolutely not being done before code > comes into master. I dislike this fact, but live with it. > I don't see why a feature has to be published when it is implemented. So I don't agree on the docs. The great advantage of hidden features is that you can take them out again more easily when they don't meet the promise they made when implemented. So Docs shouldn't be written (or at least published) until features are used in real live imho. > > What I would like to see provided by CloudStack is the ability to > > upgrade all our test environments, staging environments, UATs, what > > have you in continuous integration and have tests run on the > > upgraded setup. That allows incrementally testing CloudStack the way > > users do it. The current design of installing everything from scratch, > > redoing the testbed for each automated test run is mostly a workaround > > for that inability. If we had this ability tests written in marvin can > > be run against live setups at all times as and when features merge to > > master. > Yes and it would still be great if we could build these installations of arbitrary version as required to de upgrades from any version to any higher version and run the tests on those. For that marvin (or some tool using marvin) must be able to preinstall a setup on an version x upgrade to y and then run the required test. > It has always been my goal to reduce the barrier to writing tests for > > cloudstack and that's entirely the goal of the refactor as well. So I > > hope I have spent my time well. Since I can't be after people to write > > tests, and setup mandates, I aim to address the technical difficulty > > in writing tests through this refactor and separation. > But you do agree that making all of us write tests, unit as well as integration, is the ultimate goal, don't you. > I'll say one thing though... I'm exceptionally frustrated with the > features / commits coming into master with little to no tests (unit OR > integration). I'm at a loss at this point, and frankly considering a > possible reality that this community (generally) just doesn't care about > making it easy to reproduce tests. The lack of interest and ignorance > about the pain that this causes is infuriating to me. > Here I can't speak for all of us featurists but I want my features to work after I drop them. And I have the good sense to see that unit tests are not sufficient for that. I feel guilt as well as infuriation. Taking a side here against my wish: It is hard to learn 2 million lines of code implement a feature see it work on a version and then try to guarantee to the rest of the world that it will not break a twig down under. We should. We should also make this as easy as we can. So for now we shouldn't ever accept a submission without unit tests and strive to work towards a situation where we don't have to accept submissions without integration tests any longer. Let's be practical about it and make sure that both paths (marvin and cloudstack repo) are open and stimulate the use of both. One other thing: I refer to my self as the guilty party somehow. This is nonsense but it helps me focus on the goal. One goal is to have people outside Citrix write integration tests. I wrote integration tests for some features that where a rip-off of some integration tests that Hugo wrote for his. We run them once in a while (as in every release but not every RC) I am yet to learn the Citrix way of doing this and will focus on that. You can not blame a featurist like my self to first focus on the code to get something done. Now that o have a grasp of a few percent of those two million lines I am happy to learn more about marvin and rewrite our tests. patients with me please, I am willing, Daan, (and I am sure loads of others)
