> -----Original Message----- > From: Daan Hoogland [mailto:daan.hoogl...@gmail.com] > Sent: Saturday, October 05, 2013 3:22 AM > To: dev > Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack > > H Chip, > > As a feature-dev driven by a 150-man-strong-cloud-operator-base that is > not interested in anything but me showing that they can work with what > they asked for: > > On Fri, Oct 4, 2013 at 10:52 PM, Chip Childers > <chip.child...@sungard.com>wrote: > ... > > > On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote: > > > I'll summarize and address the concerns raised so far. > > > > > > Marvin has been in this repo for a long time for us to start writing > > > tests. The only tests I've seen coming are from a specific set of > > > people focussed on QA efforts. > > > > I agree - and that's a problem. New features should *ALL* have tests > > before they merge into master. I think that assuming that the only > > test writers are a group of folks that write the tests today is > > actually a larger problem. > > > > Yes and we will need to work down a backlog of scenarios before we ever > can rely on guys like me doing that. Not because they won't but because > there is to much to write tests for edging on the new features they > write. Just because those tests aren't there yet. I think giving Citrix > QA a repo to work on is fine but I would like to see it merged back at > some point and a continued possibility to write them in the main tree. > [Animesh>] While I don't agree to a separate repo for tests (marvin framework is ok) I do want to call out the proposal is not for giving Citrix QA a repo to work on and I don't think Prasanna meant that way.
> > > > > I want to reduce the impediment for > > > people who are writing tests *today*. Those looking to get started > > > in the near future won't have any new learning to do, just that > > > their code goes in an alternate repo that is pointed to right > > > infrastructure. > > > @prasanna: education of the developers is also something to consider. I > know I could contribute more to integration and at Schuberg Philis we > have some work going on in that direction. It would be nice to see easy > integration of this with the Citrix work. > > > > > ... > > > > > > > Reason for tests and framework to go together is simple. > > > Guys, why not both. some general tests that will be valid for any ACS > version can go in the marvin code base. Others that change over time > should be tied to the version of ACS they apply to. > > Test can be moved from one repo to the other if needed. > > ... > > > > > That's marvin, not the tests, right? If I add a new feature for 4.3, > > should that test appear in a 4.2 run? Aren't we aiming for (I know > > we're not there) a situation where *all* tests pass before we release? > > IMO not versioning the tests (not the Marvin framework) with the > > target of the tests is confusing. > > > That I agree with completely. > > > > > The infrastructure (currently) only looks at the changes in the test > > > directory before performing a run. It doesn't care whether server/ > > > was changed or plugins/x/y/z was changed. That's because the tests > > > are unrelated to what is in the rest of the repository. In fact you > > > can't even run them without a deployed cloud. So I don't see why > > > idle code should lie in the repo. > > > You can trigger the building of a cloud when code changes and run the > tests against that. > > > > > > > Integration tests are essential, they will keep coming as long as > > > Citrix QA is invested in the effort, but they need to come faster > > > into the repos and that will be addressed by the separation IMO. > > > > How? > > > > > Managing > > > the feature submitted to cloudstack against tests submitted to > > > marvin is not a hard thing to do. We simply mirror the release > > > branches in marvin and submit tests there. In fact I wonder why we > > > didn't have this question when docs were separated? It doesn't work > > > any differently really. > > > > Well, it turns out that docs are absolutely not being done before code > > comes into master. I dislike this fact, but live with it. > > > I don't see why a feature has to be published when it is implemented. So > I don't agree on the docs. The great advantage of hidden features is > that you can take them out again more easily when they don't meet the > promise they made when implemented. So Docs shouldn't be written (or at > least published) until features are used in real live imho. > > > > > What I would like to see provided by CloudStack is the ability to > > > upgrade all our test environments, staging environments, UATs, what > > > have you in continuous integration and have tests run on the > > > upgraded setup. That allows incrementally testing CloudStack the way > > > users do it. The current design of installing everything from > > > scratch, redoing the testbed for each automated test run is mostly a > > > workaround for that inability. If we had this ability tests written > > > in marvin can be run against live setups at all times as and when > > > features merge to master. > > > Yes and it would still be great if we could build these installations of > arbitrary version as required to de upgrades from any version to any > higher version and run the tests on those. For that marvin (or some tool > using > marvin) must be able to preinstall a setup on an version x upgrade to y > and then run the required test. > > > It has always been my goal to reduce the barrier to writing tests for > > > cloudstack and that's entirely the goal of the refactor as well. So > > > I hope I have spent my time well. Since I can't be after people to > > > write tests, and setup mandates, I aim to address the technical > > > difficulty in writing tests through this refactor and separation. > > > > But you do agree that making all of us write tests, unit as well as > integration, is the ultimate goal, don't you. > > > > I'll say one thing though... I'm exceptionally frustrated with the > > features / commits coming into master with little to no tests (unit OR > > integration). I'm at a loss at this point, and frankly considering a > > possible reality that this community (generally) just doesn't care > > about making it easy to reproduce tests. The lack of interest and > > ignorance about the pain that this causes is infuriating to me. > > > Here I can't speak for all of us featurists but I want my features to > work after I drop them. And I have the good sense to see that unit tests > are not sufficient for that. I feel guilt as well as infuriation. Taking > a side here against my wish: It is hard to learn 2 million lines of code > implement a feature see it work on a version and then try to guarantee > to the rest of the world that it will not break a twig down under. We > should. We should also make this as easy as we can. So for now we > shouldn't ever accept a submission without unit tests and strive to work > towards a situation where we don't have to accept submissions without > integration tests any longer. > > Let's be practical about it and make sure that both paths (marvin and > cloudstack repo) are open and stimulate the use of both. > > One other thing: I refer to my self as the guilty party somehow. This is > nonsense but it helps me focus on the goal. One goal is to have people > outside Citrix write integration tests. I wrote integration tests for > some features that where a rip-off of some integration tests that Hugo > wrote for his. We run them once in a while (as in every release but not > every RC) I am yet to learn the Citrix way of doing this and will focus > on that. You can not blame a featurist like my self to first focus on > the code to get something done. Now that o have a grasp of a few percent > of those two million lines I am happy to learn more about marvin and > rewrite our tests. > > patients with me please, I am willing, > Daan, (and I am sure loads of others)
