"Solr Orbit" seems a fine name to me. Re: David's point on picking one of search-benchmark-game vs. solr-benchmark-fork
I'll add my opinion as I've used / made code changes to both and have provided this feedback to Eric / Jan. The search-benchmark-game is nice for its simplicity and results presentation. It can quickly give you some data points between two versions of software. That being said, at best it seems geared more towards assessing a library (lucene) than a full distributed system (solr). Its feature set is limited and the methodology for executing queries / drawing statistical conclusions leaves something to be desired (can expand in detail if anyone wants to hear). In contrast, solr-benchmark (OSB fork) is building off a feature set constructed over the better part of a decade. It's a much larger codebase and a more mature benchmarking framework with much greater capability to do different kinds of performance test workloads. Personally I would go with solr-benchmark. Building a web interface on top (or hooking up a jenkins build pipeline) can be done. And like we said in the meeting, just getting people to run the tool independently and publishing their results, even as plain text on the wiki, would be a good first step win. From: [email protected] At: 04/16/26 13:00:14 UTC-4:00To: [email protected] Subject: Re: Solr Performance Benchmarking Discussion Page It's not work using Elastic Inc's trademark "Rally" if we can avoid it. They are known to pursue such violations. And I believe it is a violation of the Apache license too. I like "Solr Orbit" :) I'll wait some time to let all voices be heard wrt the various tools and which to pick going forward, then if everyone rallies (pun intended) around solr-benchmark, then we can proceed. Jan > 16. apr. 2026 kl. 14:36 skrev David Smiley <[email protected]>: > > Why avoid the word "Rally" if that's its foundation/lineage? If we > want to avoid risk of using that word, then we simply ask > ElasticSearch with our proposed name for our fork that includes that > word. > > On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <[email protected]> wrote: >> >> Hi, >> >> Thanks for presenting the solr-benchmark tool on the community meetup yesterday Kevin. >> I'll encourage others to take it for a spin. There are rough edges but you should be able to get some results. >> >> We also discussed the potential for moving the tool from my github space to asf. >> That would make it easier for the community to contribute and collaborate. But it requires 2-3 PMC members interested in being maintainers. >> If we make it an asf repo, David noted that we are not required to publish official releases until we feel compelled to do so. >> Also, David noted the naming collision with the internal solr-benchmark module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race track) but in our Solar-system universe :) >> >> As a next step I'll start a VOTE thread for accepting the two repos into ASF. The results of the VOTE will guide further action. >> >> Jan >> >>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <[email protected]>: >>> >>> Hi, >>> >>> I read through the benchmarking wiki page today. It was well written, thanks Kevin for reviving this effort, and in such a structured way! >>> >>> There's a renewed energy around benchmarking, and more tooling options than ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr Benchmark": >>> >>> https://github.com/janhoy/solr-benchmark >>> >>> You may quickly notice that this looks familiar. And yes, it is a fork/port of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters, using the same datasets and "workload"s as those tools. >>> I first tried to start a fork 2 or 3 years ago but it stranded. I made a new effort using LLM agents and this first working version was prepared in a few afternoons. >>> Even if the foundation is solid and proven over many years, this initial port is not complete, view it as a MVP and WIP. Only one workload / dataset is ported so far. Take it for a spin... >>> >>> I'll defer to Kevin to add it to the tools list of the wiki and continue the analysis effort with this as one of the contenders. >>> >>> Jan >>> >>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A) <[email protected]>: >>>> >>>> Hello everyone, >>>> >>>> Given the recent interest and discussion around Solr performance benchmarking, I figured it would be useful to 1) centralize the discussion and 2) bring it to a long-lived format (that's not email). So with that, I have started https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking (I figured there's still more discussion to be had before it becomes a SIP with technical requirements). >>>> >>>> I encourage anyone and everyone who is interested to provide their input (comment or edit). This is a community initiative, and shouldn't be limited by me or any biases I may have. Hopefully people find this useful in moving the discussion forward. >>>> >>>> -Kevin >>> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
