Hi.
One of the reasons why our CI takes so long is the old habit to execute
each of our tests in a separate JVM. I am aware (now after five years
working on GraalVM) that JVM doesn't do anything useful for the first two
seconds. It starts in interpreter mode, collects profiles, invokes C1
compiler and (in the worst) case also C2 compiler. Only then the JVM
reaches the point of "peak performance" when it is actually useful. However
before reaching the point many of our tests are over. As a result running:
$ ant -f java/java.source.base/ test
just took 9.5 minutes on my computer. The cure is simple, if I change the
forkMode:
diff --git a/nbbuild/templates/common.xml b/nbbuild/templates/common.xml
index 377b6db0d1a4..ec063c3ba7b4 100644
--- a/nbbuild/templates/common.xml
+++ b/nbbuild/templates/common.xml
@@ -740,7 +740,7 @@
<attribute name="test.type"/>
<attribute name="disable.apple.ui" default="false"/>
<sequential>
- <junit showoutput="true" fork="true"
failureproperty="tests.failed" errorproperty="tests.failed"
filtertrace="${test.filter.trace}"
tempdir="${build.test.@{test.type}.results.dir}"
timeout=
"${test.timeout}" jvm="${test.nbjdk.java}">
+ <junit showoutput="true" fork="true" forkmode="perBatch"
failureproperty="tests.failed" errorproperty="tests.failed"
filtertrace="${test.filter.trace}" tempdir="${build.test.@{test.type}.re
sults.dir}" timeout="${test.timeout}" jvm="${test.nbjdk.java}">
<batchtest todir="${build.test.@{test.type}.results.dir}">
<fileset dir="${build.test.@{test.type}.classes.dir}"
includes="${test.includes}" excludes="${test.excludes}"/>
</batchtest>
then the test suite runs in 4.5 minutes. I believe similar speed up could
be achieved in other test suites as well.
Of course, there is a drawback. By running the tests in a single JVM they
may start to influence each other. Sure, unit tests shouldn't do that, but
that is easier said than done, especially with [Injectable Singletons](
http://wiki.apidesign.org/wiki/Injectable_Singleton) - used everywhere in
NetBeans - and their limited co-existence capabilities. As such we probably
need an opt-in for modules/tests to turn the single JVM mode on.
Does making CI faster worth it? Anyone willing to give it a try?
-jt
PS: Please avoid [Pentium-like jokes](
https://twitter.com/chainq/status/948725922642976768) that it is "faster
but incorrect". With opt-in, it can be faster and correct.
