Thanks Jesse. Great write up!
On 10/3/11 4:55 PM, "Jesse Yates" <jesse.k.ya...@gmail.com> wrote: >Hey everyone, > >There has been a bunch of work recently on speeding up the testing to make >it easier for developers to iterate quickly on new features fixes. Part of >the problem is that the test suite takes anywhere from 1-2 hrs to run and >have some apparently non-deterministic hanging of tests. > >TL;DR To speed it all up, the attack plan would be: >(1) Move long running tests to be integration tests, >(2) Use a build server for patches so people only run unit tests locally, >(3) We add unit tests when integration tests are breaking a lot but unit >tests pass, >(4) Go from forked to single jvm unit tests, >(5) Add in surefire parallelization >(6) Entertain using HBaseTestingUtilFactory > >I recently chatted with Stack and Doug about ways around this. Here is >what >we came up with: > >(1) Break up long running tests from medium and short (2 mins max) tests >and >move the former to be IntegrationTests. This was based on Todd's >suggestion >in HBASE-4438. Either naming them integration tests or long running >functional tests, they would become part of the 'mvn verify' rather than >the >'mvn test' suite of tests. Starting point would be to use Doug's >spreadsheet from HBASE-4448 and when its done, the script from HBASE-4480. > >Right now, that means a LOT of tests are going to shift, but it means when >developers run 'mvn test' the amount of time spent running unit tests will >be cut down dramatically (hopefully towards the sub 10 -15 mins range) > >There is an implicit problem here: if the soon to be integration tests >capture functionality that is not covered by unit tests, then people may >incorrectly think that they are not breaking things. Therefore, we would >do >(2) and (3): > >(2) Add a patch continuous integration server that goes and actually >builds >and tests patches as they come in. This would run 'mvn verify' and ensure >that the patch actually isn't breaking high level/complex functionality. >It >would be a requirement before patches are committed that they pass this >build. > >(3) If we find that the unit tests aren't covering a certain level of >functionality that is constantly breaking on the build server, we add more >unit tests of the breaking functionality to ensure the unit tests are more >complete and provide more assurances to developers when running them. > >This would be an ongoing process of comparing the integration tests vs. >the >unit tests. > >(4) Once we have a true unit test suite, we should be able to go from >'forked' jvm mode back to a single jvm for running tests. Unit tests >should >not do crazy fault injection, full failure scenarios, so they should be >able >to cleanup after themselves. This means we are going to get some speedup >from not spinning up a new jvm for each test. > >(5) Once we are running in non-forked mode, we can try turning on >parallelized test execution in surefire, which does parallel runs in a >single jvm. > >(6) Once things all run in a single jvm, using the HBaseTestUtilFactory >(HBASE-4448) make sense reuse the mini clusters across tests. > >What does everyone think of this approach? > >Thanks! > >-Jesse Yates
