It seems that over recent weeks our precommits have gotten somewhat flaky. Some of this is due to actual flaky tests (most of which are tracked by JIRAs) but a lot has been due to issues like clock synchronization problems on the dist-test slaves.
I'd like to consider changing precommit to retry _all_ tests up to 3 times, instead of just known-flakies. It's a bit of a heavy hammer -- the risk is that if you introduce flakiness in a test you aren't likely to see it precommit, but I think the upside of avoiding wasted effort triaging failed precommits is probably worth it. Longer term hopefully we can improve the dist-test software to support something like a "retry if results match a certain regex" to check for clock sync errors or somesuch, but I think it's non-trivial. Thoughts? -Todd -- Todd Lipcon Software Engineer, Cloudera
