A big THANK YOU for working on this Ishan! Can we add regular benchmarks based on /solr/benchmark that Mark contributed not too long ago?
~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Wed, Nov 9, 2022 at 1:39 AM Ishan Chattopadhyaya < [email protected]> wrote: > I'm working on automating performance testing, details in > https://issues.apache.org/jira/browse/SOLR-16525. > > Even before I could complete the automation, I observed massive slowdown in > restart performance, now attributable to > https://issues.apache.org/jira/browse/SOLR-16414. This affected 9.1 > release > candidate RC1, but is now fixed in 9.1 and 9x branches. > > However, while performance was back to original levels on 9.1 branch, there > was a 80-100% slowdown on the 9x branch even after this fix. > Please see: http://mostly.cool/cluster-test.json.html > The test is here: > > https://github.com/fullstorydev/solr-bench/blob/ishan/repeatable-jenkins/suites/cluster-test.json > > In order to investigate the slowdown, I retroactively applied the patch > that fixed the performance problem in SOLR-16414 (removing use of > parallelStream) to the intermediate commits and plotted the graph: > http://mostly.cool/cluster-test-with-patch.html > > And now, two more commits with potential slowdowns are observed. Here are > the JIRA issues I've opened for both: > https://issues.apache.org/jira/browse/SOLR-16530 > https://issues.apache.org/jira/browse/SOLR-16531 > > In a week of working on this automation, I was able to catch 3 slowdowns on > the first thing I automated. It might be good to keep this running and test > other aspects. Going forward, I'll be automating more performance suites > and open blocker JIRA issues on significant performance degradation, > whenever observed. I'll make it easy for all of us to add suites to the > framework and have their personal branches/PRs tested through this. > > Please let me know about any thoughts / concerns / suggestions. > > Thanks, > Ishan >
