I see a lot of discussion on enabling CI. Agree that it is important but CI in itself is not going to solve the quality issue. The other important aspect is the effectiveness of tests that get executed as part of CI. Unless there is a proper set of automated tests (which is kept up-to-date) for each component, no system can ensure quality. This will also ease the review process, every review request should have the test results along with the patch/merge request.
Irrespective of what framework we decide to chose, the quality issue can only be solved by ensuring that automated tests are effective and properly maintained. -Koushik -----Original Message----- From: ilya musayev [mailto:ilya.mailing.li...@gmail.com] Sent: Wednesday, 20 August 2014 5:22 AM To: dev@cloudstack.apache.org Subject: Re: 4.5 RM David Since CI has been mentioned several times, do you know where we stand with donated hardware? Thanks ilya On 8/19/14, 10:15 AM, David Nalley wrote: >> IMHO we should not even release 4.5 until we have a agreed upon: >> >> -what our issues are and why we released 4.4 and 4.3 late. >> -taken action to resolve those issues -guarantees that 4.5 will be on >> time >> >> If we don't do that, I don't even know why we are putting ourselves through >> the pain of a release schedule. >> > So I've been trying to give this some thought. Here's my current line > of thinking. > > The issues with late releases are not a function of our release > process per se; but are instead a function of our development process. > CloudStack is a relatively large codebase. It has a lots of points > that interact with each other, and it's moderately complex. > Development moves forward and at least happy-path testing is done for > new features, but the range of options is so large that testing > everything is a bit difficult. When someone makes a merge request; I > suspect few people do much looking. Understandable, it's a boring > task; and really looking doesn't tell us much except for style and > egregious errors. We've rarely done mandatory testing of feature > branches before they are merged in. If you want to ship on time, you > must ensure that we are vociferously guarding the quality of the > master and release branches; that we can verify programmatically that > a commit or merge doesn't break things. We must insist on automated > testing being added. > > So I've said all of that to say that I think that ship has sailed for > 4.5. We are well past feature freeze; and we didn't really have any > gating functionality. We frankly have very little idea of quality of > whats in master right now. It's certainly worse than 4.4. So now we'll > enter code freeze, we'll try and play catch up and fix all of the > things we discover that are broken. And invariably, we'll be late > again. > > If you want to solve this problem; my personal belief is that its > really is tied to CI. Efforts around Travis are interesting and > perhaps are a piece of that puzzle. Discussions around running CI are > important as well, but I truly believe that we need a gating function > that prohibits commits that increase our level of untested code or > code that fails to pass testing. I've seen some other projects using > pull requests in github, and then using the github pull request > builder[1] for jenkins to verify that every PR works. I know we've > talked about gerrit previously, and perhaps that will work as well. > > [1] > https://wiki.cloudbees.com/bin/view/DEV/Github+Pull+Request+Validation
