Hi, Claes, Thank you for the great guideline of micro benchmarking. now I have a clearer picture for it. I'm happy to learn that 5-10 minutes for each Benchmark! Let me polish mine.
Thanks, --lx On 10/27/20, 3:59 AM, "Claes Redestad" <[email protected]> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi lx, the need to add --enable-preview is known, and it's a bug introduced a couple of months ago thanks to a suggestion I did (without realizing that compiling one micro with --enable-preview would poison all of them..). This issue is tracked by a few bugs, see https://bugs.openjdk.java.net/browse/JDK-8250669 and https://bugs.openjdk.java.net/browse/JDK-8253828 The fix will likely be to remove the --enable-preview from the micros build after Records goes out of preview and then think more carefully on how to add support for benchmarking preview features. As to your other question: When adding a microbenchmark it's appreciated if the default configuration makes it as quick to run as possible without sacrificing quality. Often the default number of forks, iteration runtime et.c. can be tuned down by using custom settings in @Fork et.c. Custom parameters used can be limited to a select handful. Still it's normal for a "good" microbenchmark to require at least five, usually ten minutes per @Benchmark to produce reasonably trustworthy results. I can't speak for what others do, but we (Oracle) select what to run regularly from the available microbenchmarks at our discretion, though. That means adding a microbenchmark that takes a prohibitively long time won't be disruptive to our nightly testing, but for obvious (mostly economic) reasons we'd be unlikely to add an extremely long-running micro to regular coverage. /Claes On 2020-10-27 10:06, Liu, Xin wrote: > Hi, > > I follow the instruction on https://htmlpreview.github.io/?https://github.com/openjdk/jdk/blob/master/doc/testing.html#microbenchmarks to run a new microbenchmark. > I got error message as follows. > java.lang.UnsupportedClassVersionError: Preview features are not enabled for org/openjdk/bench/java/lang/reflect/proxy/jmh_generated/ProxyBench_newProxyInstance1i_jmhTest (class file version 60.65535). Try running with '--enable-preview' > at java.base/java.lang.ClassLoader.defineClass1(Native Method) > at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1010) > at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150) > at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:855) > at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:753) > at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:676) > at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:634) > at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182) > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519) > at java.base/java.lang.Class.forName0(Native Method) > at java.base/java.lang.Class.forName(Class.java:377) > at org.openjdk.jmh.util.ClassUtils.loadClass(ClassUtils.java:72) > at org.openjdk.jmh.runner.BenchmarkHandler.<init>(BenchmarkHandler.java:68) > at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:232) > at org.openjdk.jmh.runner.BaseRunner.doSingle(BaseRunner.java:138) > at org.openjdk.jmh.runner.BaseRunner.runBenchmarksForked(BaseRunner.java:75) > at org.openjdk.jmh.runner.ForkedRunner.run(ForkedRunner.java:72) > at org.openjdk.jmh.runner.ForkedMain.main(ForkedMain.java:84) > > > If I add an extra flag “—enable-preview” , the micro bench will work. > make test TEST="micro:java.lang.reflect" MICRO="VM_OPTIONS=--enable-preview" > > I am using jmh from make/devkit/createJMHBundle.sh, which gives me jmh-core-1.26.jar > Is that jmh-core too old or we should adjust RunTest.gmk a little bit to have that flag? > > Does Openjdk CI run benchmarks to detect performance regression? If so, could you point me to a guideline of microbenmark? > I lack of the basic sense of a “good” microbenchmark. My concern is the new micro I added is too long to break the established test run. > > Thanks, > --lx > >
