On Wed, Aug 3, 2016 at 5:47 PM Sean Busbey <bus...@cloudera.com> wrote:
> My understanding was that maintenance releases (aka double dot, e.g. > 1.7.2) had relaxed criteria because we expected the scope of changes > in them to be more limited. Even so, the release notes for 1.7.2, > 1.7.1, and 1.7.0 all claim the ITs passed. > > Even those releases have periodic IT failure. > Is there a reason we can't parallelize the ITs? We can. Eric's mrit effort was all intended towards that. But, that's not the same as CI passing. I don't know what it would take to parallelize them in a CI server. > What's stopping > builds.a.o from running them? Specific requests from projects to asf > infra can get us resources if that's the problem. > > I spoke to infra in HipChat about this a a few weeks ago, and mentioned a few things which impact builds on ASF jenkins (builds.apache.org): 1. Accumulo has an excessive number of tests to run. 2. Build timeouts with Jenkins can abort builds. 3. Tests are timing sensitive, and are affected by VM/host configuration and contention with other concurrent builds from other projects. 4. Tests need lots of RAM and storage (at least 4GB RAM, but ideally no less than 16GB, and at least 6 GB for a workspace) 5. Tests need specialized system configuration, (increasing ulimits, optimizing kernel settings for swappiness, etc.) What we really need for reliable IT passing in CI, is exclusive use of dedicated, bare-metal beefy build machines, for 6+ hours per build x 4 branches minimum, plus another 6+ hours for each pull request and other builds which skipITs, so we can get immediate feedback on unit tests and compilation errors. I don't think it's reasonable for us to even ask for such resources from INFRA. Using ASF Jenkins for -DskipITs I think is reasonable, with individual testing of full IT suite from the developers/contributors, as well as feedback from independent dedicated CI builds (such as the Jenkins server I'm running, as well as any Josh might still be running, and others). The minimum passing to create a release candidate is -Psunny. Usually folks submit test results on the full ITs which informed their vote. Sometimes, but not always, the release manager will do some pre-vote testing and prepare draft release notes. Testing information also gets aggregated from the vote thread onto the release notes.
