I know this was brought up almost a year ago, but it might be time to revisit breaking out the tests into an integration test suite and a regular unit test suite. We can use the the maven failsafe plugin ( http://maven.apache.org/plugins/maven-failsafe-plugin/) which is designed to run integration tests, like surefire is built to run unit tests.
The advantage of breaking them out into two different sets of tests would be that developers can run tests hierarchically, rather than in bulk. So if you make a change, first you run the unit tests. If those pass (and unit tests should really max at 10mins, tops), then go on to run the integration tests, which actually build full instances of mapreduce/hdfs/zookeeper. This will help people iterate more quickly on broken pieces but still be 'sure' that the system still works given a pass w/ the integration tests. To help with this refactoring, we would still want to move to more interfaces w/ mocking as Stack suggests so the unit tests can be pretty comprehensive. However, there really isn't a substitute for running against 'real' objects. I'd be happy to help on this transition, if the we decide to go this route. As for a 'barney', maybe something through twitter where when someone breaks the build, the previous person posts a pic of barney to the receiver? -Jesse Yates On Sun, Sep 18, 2011 at 12:28 PM, Stack <[email protected]> wrote: > On Fri, Sep 16, 2011 at 11:39 PM, Akash Ashok <[email protected]> > wrote: > > Firstly I end up running the whole suite without a the patch and second > > with patch and compare the results. But it turns out most of the times > > there's always inconsistency. Meaning to say a few tests are erratic. The > > fail while running the whole suite and pass when run alone. Moreover > running > > the whole test suite takes about 2 hours. > > > > This is taking a lot of time for me to work on the simplest of the > patches. > > > > Yes, this is a problem. That the tests take so long to run, I'm sure, > is a barrier to contribution. It also in part explains why our test > builds are so often red -- because its onerous running full suite. We > used to huff and puff about the fact that our test suite took an hour. > I hadn't realized we were beyond the two hour mark now. Need to work > on this. Will file some issues for the longer-running tests. Some we > can aggregate. Whats really needed are better tools and more use of > Interfaces so the parts not under test can be mocked rather than have > to stand up real instances of an hdfs or mapreduce cluster. Need to > work on this too. > > That our test suite is seen to be flakey doesn't inspire confidence > and makes it so you do the crazy thing of having to run once to see > what the current state is before you apply your patch (two hours > later). This should start to improve at least on the branch when we > start to work on stabilization. > > At a place I used to work, if you broke the build you were given a > giant purple barney -- http://www.barney.com/usa/index.asp -- and you > had to wear it on your desk until someone else broke the build and > then you could pass it off. The contortions fellas went through to > avoid being barney-ified were monstrous. We need something like a > virtual barney for hbase. Suggestions? > > St.Ack >
