Thank you for sending these detailed notes. I fear I will be duplicating these efforts on branch-1 when (eventually) preparing for 1.7.0.
On Fri, May 22, 2020 at 1:32 PM Stack <st...@duboce.net> wrote: > After a bit of work, there are currently no flakies in branch-2.3 and all > tests passed over the last ten nightlies (a nightly is a comprehensive > build that runs the full test suite once for jdk8+hadoop2, again for > jdk8+hadoop3, and again for jdk11+hadoop3). You can see this by looking at > our flakies dashboard for branch-2.3 [1][2]. Branch-2 is not too far behind > with one flakey and a recent nightly test failure [3]. > > This 'cleanliness' is a little noteworthy, IMO. > > Other branches have not had the same focus so their state varies w/ > attention paid. > > Attempts were also recently made at speeding up the jenkins test builds > playing w/ maven forkcount, shrinking test resource usage, and with the > maven -T which allows manipulating levels of maven module build/test > parallelism (HBASE-24150, HBASE-24072, etc.). There was little yield to be > had here...perhaps a 20% improvement. Complications included: jenkins build > slaves allow two executors/builds to run at the same time so when an hbase > build runs, it is sharing the machine w/ another (often another hbase > build); host and docker resource constraints; and that our module > inter-dependency constrains how much parallelism is allowed. > > As part of the above work in branch-2/branch-2.3, tests were run locally on > various hardware. It should come as no surprise that the experience varied > w/ environment (less so as flakies were addressed). On better hardware, > tests can be made run more furiously so they use all the machine and > complete faster. > > The settings we have as our defaults are configured to suit the Apache > Jenkins build environment which is usually 16CPUs/48G. As said above, > Jenkins slaves allow two builds machines so halve these resources when an > HBase build runs on Apache Infrastructure. So as to be considerate of our > companion Apache projects, defaults are relatively 'mild': our forkcount is > set to 0.25 all of the CPUs in the machine. On Apache Jenkins, 0.25*16CPU > == 4 CPUs for hbase build. We also set -T2 which means up to two modules > building in parallel where possible (each with above configured forkcount). > Our test suites on Jenkins continue to take hours. > > On a 40CPU linux machine with the below arguments where we use half the > CPUs in the machine (and ulimit -u 40960), all tests run in just under an > hour: > > $ x="0.50C" ; nohup mvn -T2 -Dsurefire.firstPartForkCount=$x > -Dsurefire.secondPartForkCount=$x test -PrunAllTests > > Upping the forkcount on this machine beyond 0.50C tended to bring a rush of > tests exiting... (To be investigated). On this machine, tests currently > pass about 80% of the time. To be improved. > > On an anemic 4CPU VM, I can run the below and it will pass 60% of the time. > It takes ~5hours: > > $ x="1.0C" ; nohup mvn -T2 -Dsurefire.firstPartForkCount=$x > -Dsurefire.secondPartForkCount=$x test -PrunAllTests > > On a mac w/ 12CPUs, I can run same command as above. It passes with about > same frequency and takes just over 1 1/2 hours. > > On my laptop it is less reliable passing about 1/3rd of the time in about 2 > 1/2 hours. > > If I use less resources, a lesser forkcount, the tests complete more often > (but take correspondingly longer). > > Going forward, we will continue to watch branch-2/branch-2.3. Regards > speedup, there is a bunch to do. A large win is to be had improving the > HDFS mini cluster adding configuration (lots of resources such as pool > thread counts are hard coded and numbers that are large for small test run) > and working on speeding startup times. > > oao, > S > > 1. > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.3/lastSuccessfulBuild/artifact/dashboard.html > 2. Unfortunately, the nightly list shows reds though all tests passed > because of report assemblage issues being addressed by infra: > https://issues.apache.org/jira/browse/INFRA-20025 > 3. > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
