[ https://issues.apache.org/jira/browse/SOLR-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350128#comment-17350128 ]
Mark Robert Miller edited comment on SOLR-15428 at 5/23/21, 8:02 PM: --------------------------------------------------------------------- Implementation is not currently based on the JMH Gradle Plugin with fat jars, shading, it's own config setup, etc, but a more direct implementation into raw jmh via a JavaExec task. Simple example usage and output (via latest local ref branch, my 'apache/main' work on this is not fully synced up yet) The following would kick off a full integration type test called PointsVsTrieIndex, utilizing MiniSolrCloudCluster and multiple shards + replicas. Ideally, much like good Unit Testing, it would be augmented with additional microbenchmarks at much lower levels that do not spin up clusters and Jetty instances, etc. The implementation (not currently included) is standard jmh annotated benchmark class Java code. {noformat} /data2/lucene-solr-3/solr/test-framework$ gw --no-daemon jmh -Pinclude=org.apache.solr.bench.schema.PointsVsTrieIndex.* ... # JMH version: 1.29 # VM version: JDK 11.0.11, OpenJDK 64-Bit Server VM, 11.0.11+9-Ubuntu-0ubuntu2 # VM invoker: /usr/lib/jvm/java-11-openjdk-amd64/bin/java # VM options: -Xms4g -Djmh.separateClassLoader=true -Dlog4j2.is.webapp=false -Dlog4j2.garbagefreeThreadContextMap=true -Dlog4j2.enableDirectEncoders=true -Dlog4j2.enable.threadlocals=true -XX:ConcGCThreads=2 -XX:ParallelGCThreads=3 -XX:+UseG1GC -Djetty.insecurerandom=1 -Djava.security.egd=file:/dev/./urandom -XX:-UseBiasedLocking -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -Dsolr.log.dir= -Djava.class.path=... -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Blackhole mode: full + dont-inline hint # Warmup: 4 iterations, 10 s each # Measurement: 5 iterations, 10 s each # Timeout: 300 s per iteration # Threads: 6 threads, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.solr.bench.schema.PointsVsTrieIndex.indexSmallDoc # Parameters: (docValues = useAsStored, fieldType = trie) # Run progress: 83.33% complete, ETA 00:01:45 # Fork: 1 of 1 # Warmup Iteration 1: cluster base path=/tmp/test-1_001-948879960720677/tempDir-001 4.062 ops/ms # Warmup Iteration 2: cluster base path=/tmp/test-1_001-948879960720677/tempDir-002 4.890 ops/ms # Warmup Iteration 3: cluster base path=/tmp/test-1_001-948879960720677/tempDir-003 5.503 ops/ms # Warmup Iteration 4: cluster base path=/tmp/test-1_001-948879960720677/tempDir-004 5.438 ops/ms Iteration 1: cluster base path=/tmp/test-1_001-948879960720677/tempDir-005 <5.378 ops/ms Iteration 2: cluster base path=/tmp/test-1_001-948879960720677/tempDir-006 5.541 ops/ms Iteration 3: cluster base path=/tmp/test-1_001-948879960720677/tempDir-007 5.586 ops/ms Iteration 4: cluster base path=/tmp/test-1_001-948879960720677/tempDir-008 5.523 ops/ms Iteration 5: cluster base path=/tmp/test-1_001-948879960720677/tempDir-009 5.517 ops/ms Result "org.apache.solr.bench.schema.PointsVsTrieIndex.indexSmallDoc": 5.509 ±(99.9%) 0.301 ops/ms [Average] (min, avg, max) = (5.378, 5.509, 5.586), stdev = 0.078 CI (99.9%): [5.208, 5.810] (assumes normal distribution) # Run complete. Total time: 00:10:35 REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial experiments, perform baseline and negative tests that provide experimental control, make sure the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts. Do not assume the numbers tell you what you want them to tell. Benchmark (docValues) (fieldType) Mode Cnt Score Error Units PointsVsTrieIndex.indexSmallDoc disabled point thrpt 5 5.927 ± 0.192 ops/ms PointsVsTrieIndex.indexSmallDoc disabled trie thrpt 5 5.559 ± 0.180 ops/ms PointsVsTrieIndex.indexSmallDoc enabled point thrpt 5 5.776 ± 1.263 ops/ms PointsVsTrieIndex.indexSmallDoc enabled trie thrpt 5 5.681 ± 0.264 ops/ms PointsVsTrieIndex.indexSmallDoc useAsStored point thrpt 5 5.884 ± 0.211 ops/ms PointsVsTrieIndex.indexSmallDoc useAsStored trie thrpt 5 5.509 ± 0.301 ops/ms Benchmark result is saved to /data2/lucene-solr-3/solr/test-framework/build/reports/jmh/result.json {noformat} Result format can be adjusted as desired as pretty much all options provided by jmh ([https://openjdk.java.net/projects/code-tools/jmh/)] For graphing, or comparing a run before and after your changes, I would recommend the excellent JMH Visualizer, [https://jmh.morethan.io/.|https://jmh.morethan.io/] Easy to use in a variety of ways via that site and easy to run locally via GitHub source. was (Author: markrmiller): Implementation is not currently based on the JMH Gradle Plugin with fat jars, shading, it's own config setup, etc, but a more direct implementation into raw jmh via a JavaExec task. Simple example usage and output (via latest local ref branch, my 'apache/main' work on this is not fully synced up yet) The following would kick off a full integration type test called PointsVsTrieIndex, utilizing MiniSolrCloudCluster and multiple shards + replicas. Ideally, much like good Unit Testing, it would be augmented with additional microbenchmarks at much lower levels that do not spin up clusters and Jetty instances, etc. The implementation (not currently included) is standard jmh annotated benchmark class Java code. {noformat} /data2/lucene-solr-3/solr/test-framework$ gw --no-daemon jmh -Pinclude=org.apache.solr.bench.schema.PointsVsTrieIndex.* ... # JMH version: 1.29 # VM version: JDK 11.0.11, OpenJDK 64-Bit Server VM, 11.0.11+9-Ubuntu-0ubuntu2 # VM invoker: /usr/lib/jvm/java-11-openjdk-amd64/bin/java # VM options: -Xms4g -Djmh.separateClassLoader=true -Dlog4j2.is.webapp=false -Dlog4j2.garbagefreeThreadContextMap=true -Dlog4j2.enableDirectEncoders=true -Dlog4j2.enable.threadlocals=true -XX:ConcGCThreads=2 -XX:ParallelGCThreads=3 -XX:+UseG1GC -Djetty.insecurerandom=1 -Djava.security.egd=file:/dev/./urandom -XX:-UseBiasedLocking -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -Dsolr.log.dir= -Djava.class.path=... -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Blackhole mode: full + dont-inline hint # Warmup: 4 iterations, 10 s each # Measurement: 5 iterations, 10 s each # Timeout: 300 s per iteration # Threads: 6 threads, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.solr.bench.schema.PointsVsTrieIndex.indexSmallDoc # Parameters: (docValues = useAsStored, fieldType = trie)# Run progress: 83.33% complete, ETA 00:01:45 # Fork: 1 of 1 # Warmup Iteration 1: cluster base path=/tmp/test-1_001-948879960720677/tempDir-001 4.062 ops/ms # Warmup Iteration 2: cluster base path=/tmp/test-1_001-948879960720677/tempDir-002 4.890 ops/ms # Warmup Iteration 3: cluster base path=/tmp/test-1_001-948879960720677/tempDir-003 5.503 ops/ms # Warmup Iteration 4: cluster base path=/tmp/test-1_001-948879960720677/tempDir-004 5.438 ops/ms Iteration 1: cluster base path=/tmp/test-1_001-948879960720677/tempDir-005 <5.378 ops/ms Iteration 2: cluster base path=/tmp/test-1_001-948879960720677/tempDir-006 5.541 ops/ms Iteration 3: cluster base path=/tmp/test-1_001-948879960720677/tempDir-007 5.586 ops/ms Iteration 4: cluster base path=/tmp/test-1_001-948879960720677/tempDir-008 5.523 ops/ms Iteration 5: cluster base path=/tmp/test-1_001-948879960720677/tempDir-009 5.517 ops/ms Result "org.apache.solr.bench.schema.PointsVsTrieIndex.indexSmallDoc": 5.509 ±(99.9%) 0.301 ops/ms [Average] (min, avg, max) = (5.378, 5.509, 5.586), stdev = 0.078 CI (99.9%): [5.208, 5.810] (assumes normal distribution) # Run complete. Total time: 00:10:35 REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial experiments, perform baseline and negative tests that provide experimental control, make sure the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts. Do not assume the numbers tell you what you want them to tell. Benchmark (docValues) (fieldType) Mode Cnt Score Error Units PointsVsTrieIndex.indexSmallDoc disabled point thrpt 5 5.927 ± 0.192 ops/ms PointsVsTrieIndex.indexSmallDoc disabled trie thrpt 5 5.559 ± 0.180 ops/ms PointsVsTrieIndex.indexSmallDoc enabled point thrpt 5 5.776 ± 1.263 ops/ms PointsVsTrieIndex.indexSmallDoc enabled trie thrpt 5 5.681 ± 0.264 ops/ms PointsVsTrieIndex.indexSmallDoc useAsStored point thrpt 5 5.884 ± 0.211 ops/ms PointsVsTrieIndex.indexSmallDoc useAsStored trie thrpt 5 5.509 ± 0.301 ops/ms Benchmark result is saved to /data2/lucene-solr-3/solr/test-framework/build/reports/jmh/result.json {noformat} Result format can be adjusted as desired as pretty much all options provided by jmh ([https://openjdk.java.net/projects/code-tools/jmh/)] For graphing, or comparing a run before and after your changes, I would recommend the excellent JMH Visualizer, [https://jmh.morethan.io/.|https://jmh.morethan.io/] Easy to use in a variety of ways via that site and easy to run locally via GitHub source. > Integrate the OpenJDK JMH micro benchmark framework for micro benchmarks and > performance comparisons and investigation. > ----------------------------------------------------------------------------------------------------------------------- > > Key: SOLR-15428 > URL: https://issues.apache.org/jira/browse/SOLR-15428 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Reporter: Mark Robert Miller > Priority: Major > > I’ve spent a fair amount of time over the years on work around integrating > Lucene’s benchmark framework into Solr and while I’ve used this with > additional local work off and on, JMH has become somewhat of a standard for > micro benchmarks on the JVM. I have some work that provides an initial > integration, allowing for more targeted micro benchmarks as well as more > integration type benchmarking using JettySolrRunner. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org