[
https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049655#comment-16049655
]
Michael Sun commented on SOLR-10317:
------------------------------------
Just uploaded the first cut of Solr benchmark I built during my work, as one
more option for community for benchmarking. There are a few good benchmarks in
the community for different use cases, using different frameworks. The goal of
my benchmark, in short, is to design an extensible, standardized benchmark that
can be used for a variety of common performance use cases. Nightly performance
regression tests are very important. Meanwhile it would be good if we can reuse
the same benchmark for capacity planning, scalability study, troubleshooting,
etc., which has slightly different requirement to nightly tests. It would be a
good saving for everyone in community if he only needs to extend the benchmark,
not rebuild one, for his own use cases in near future.
In addition, the benchmark includes a variety of instruments to help understand
why the performance is, in addition to what the performance is. One obvious
reason is that answering why is the primary goal for some use cases, such as
troubleshooting, scalability study. Meanwhile it also helps to build 'correct'
performance tests. For example, performance bottleneck discovered in tests may
not be a code defect but some setup issue. Being able to analyze a bit can make
sure the performance tests are testing the right thing.
Designing a good benchmark is one of my primary jobs at work. So I will
continue to elaborate the framework and add new tests. There are a few good
benchmarks for Solr. Also [~vivek.nar...@uga.edu] has done a great job in
designing a few new test cases. I can help you in porting or adding new test
cases with my framework if you like.
The patch mainly includes object model and a sample test to demonstrate object
model. More components will follow. It's an option to community of course but I
do think community can benefit from this contribution. Any feedback is
appreciated.
> Solr Nightly Benchmarks
> -----------------------
>
> Key: SOLR-10317
> URL: https://issues.apache.org/jira/browse/SOLR-10317
> Project: Solr
> Issue Type: Task
> Reporter: Ishan Chattopadhyaya
> Labels: gsoc2017, mentor
> Attachments: changes-lucene-20160907.json,
> changes-solr-20160907.json, managed-schema,
> Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks.docx,
> Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks-FINAL-PROPOSAL.pdf,
> solrconfig.xml
>
>
> Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be
> found here, https://home.apache.org/~mikemccand/lucenebench/.
> Preferably, we need:
> # A suite of benchmarks that build Solr from a commit point, start Solr
> nodes, both in SolrCloud and standalone mode, and record timing information
> of various operations like indexing, querying, faceting, grouping,
> replication etc.
> # It should be possible to run them either as an independent suite or as a
> Jenkins job, and we should be able to report timings as graphs (Jenkins has
> some charting plugins).
> # The code should eventually be integrated in the Solr codebase, so that it
> never goes out of date.
> There is some prior work / discussion:
> # https://github.com/shalinmangar/solr-perf-tools (Shalin)
> # https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md
> (Ishan/Vivek)
> # SOLR-2646 & SOLR-9863 (Mark Miller)
> # https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)
> # https://github.com/lucidworks/solr-scale-tk (Tim Potter)
> There is support for building, starting, indexing/querying and stopping Solr
> in some of these frameworks above. However, the benchmarks run are very
> limited. Any of these can be a starting point, or a new framework can as well
> be used. The motivation is to be able to cover every functionality of Solr
> with a corresponding benchmark that is run every night.
> Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure
> [~shalinmangar] and [~markrmil...@gmail.com] would help here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
