Hi,
Since most tests in the Mesos, libprocess, and stout test suites can
be executed in parallel (the exception being some `ROOT` tests with
global side effects in Mesos), we recently added a parallel test
runner `support/mesos-gtest-runner.py`. This should allow to
potentially significantly speed up running of test suites.
To enable automatic parallel execution of tests for test targets
executed during `make check`, configure Mesos with the option
`--enable-parallel-test-execution`. This will configure the test runner
to run all tests but the `ROOT` tests in parallel; `ROOT` tests will
be run in a separate, sequential step.
* * *
We use the environment variable `TEST_DRIVER` to drive parallel test
execution. By setting this variable to an empty string you can
temporarily disable configured parallel execution, e.g.,
% make check TEST_DRIVER=
By setting this environment variable you have control over the test
runner itself and its arguments, even without enabling parallel test
during `./configure` time. Be aware that many `ROOT` tests cannot be
run in parallel.
The current settings oversubscribe the machine by running `#cores*1.5`
parallel jobs. This was driven by the observation that currently our
tests by and large do not make extended use of even a single core.
The number of parallel jobs can by controlled with the `-j` flag of
the test runner.
Since making more use of the machine will likely increase machine load
during test execution, running tests in parallel might expose test
flakiness. Tests might also fail to run in parallel if testcases e.g.,
write data to hardcoded locations or use hardcoded ports. Please file
JIRA tickets for such tests if they do not yet exist.
There is still some work needed to improve reporting from parallel
tests. We currently use a very silent mode if tests are running
without failures, and just report the logs of failed jobs in case of
failure. MESOS-6387 sketches out possible future improvements in this
area.
Happy testing,
Benjamin with help from Kevin & Till
