Home after some travel... I think the solr-benchmark-game needs a bit more marinating to see the true value. Also, Kevin and (???) did some great analysis on it and pointed out some big weaknesses to it's approach.
I'm onboard with the solr-orbit plan, and can see how I would use it with my client to establish the benchmarks needed. On 2026/04/17 20:38:17 "Kevin Liang (BLOOMBERG/ 919 3RD A) via dev" wrote: > "Solr Orbit" seems a fine name to me. > > Re: David's point on picking one of search-benchmark-game vs. > solr-benchmark-fork > > I'll add my opinion as I've used / made code changes to both and have > provided this feedback to Eric / Jan. > > The search-benchmark-game is nice for its simplicity and results > presentation. It can quickly give you some data points between two versions > of software. That being said, at best it seems geared more towards assessing > a library (lucene) than a full distributed system (solr). Its feature set is > limited and the methodology for executing queries / drawing statistical > conclusions leaves something to be desired (can expand in detail if anyone > wants to hear). > > In contrast, solr-benchmark (OSB fork) is building off a feature set > constructed over the better part of a decade. It's a much larger codebase and > a more mature benchmarking framework with much greater capability to do > different kinds of performance test workloads. > > Personally I would go with solr-benchmark. Building a web interface on top > (or hooking up a jenkins build pipeline) can be done. And like we said in the > meeting, just getting people to run the tool independently and publishing > their results, even as plain text on the wiki, would be a good first step > win. > > From: [email protected] At: 04/16/26 13:00:14 UTC-4:00To: > [email protected] > Subject: Re: Solr Performance Benchmarking Discussion Page > > It's not work using Elastic Inc's trademark "Rally" if we can avoid it. They > are known to pursue such violations. And I believe it is a violation of the > Apache license too. I like "Solr Orbit" :) > > I'll wait some time to let all voices be heard wrt the various tools and > which > to pick going forward, then if everyone rallies (pun intended) around > solr-benchmark, then we can proceed. > > Jan > > > 16. apr. 2026 kl. 14:36 skrev David Smiley <[email protected]>: > > > > Why avoid the word "Rally" if that's its foundation/lineage? If we > > want to avoid risk of using that word, then we simply ask > > ElasticSearch with our proposed name for our fork that includes that > > word. > > > > On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <[email protected]> wrote: > >> > >> Hi, > >> > >> Thanks for presenting the solr-benchmark tool on the community meetup > yesterday Kevin. > >> I'll encourage others to take it for a spin. There are rough edges but you > should be able to get some results. > >> > >> We also discussed the potential for moving the tool from my github space > >> to > asf. > >> That would make it easier for the community to contribute and collaborate. > But it requires 2-3 PMC members interested in being maintainers. > >> If we make it an asf repo, David noted that we are not required to publish > official releases until we feel compelled to do so. > >> Also, David noted the naming collision with the internal solr-benchmark > module. How about "solr-orbit", which is a homage to "Rally" (orbiting the > race > track) but in our Solar-system universe :) > >> > >> As a next step I'll start a VOTE thread for accepting the two repos into > ASF. The results of the VOTE will guide further action. > >> > >> Jan > >> > >>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <[email protected]>: > >>> > >>> Hi, > >>> > >>> I read through the benchmarking wiki page today. It was well written, > thanks Kevin for reviving this effort, and in such a structured way! > >>> > >>> There's a renewed energy around benchmarking, and more tooling options > >>> than > ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr > Benchmark": > >>> > >>> https://github.com/janhoy/solr-benchmark > >>> > >>> You may quickly notice that this looks familiar. And yes, it is a > >>> fork/port > of Rally/OpensearchBenchmark, ported to provision and benchmark Solr > clusters, > using the same datasets and "workload"s as those tools. > >>> I first tried to start a fork 2 or 3 years ago but it stranded. I made a > new effort using LLM agents and this first working version was prepared in a > few afternoons. > >>> Even if the foundation is solid and proven over many years, this initial > port is not complete, view it as a MVP and WIP. Only one workload / dataset > is > ported so far. Take it for a spin... > >>> > >>> I'll defer to Kevin to add it to the tools list of the wiki and continue > the analysis effort with this as one of the contenders. > >>> > >>> Jan > >>> > >>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A) > <[email protected]>: > >>>> > >>>> Hello everyone, > >>>> > >>>> Given the recent interest and discussion around Solr performance > benchmarking, I figured it would be useful to 1) centralize the discussion > and > 2) bring it to a long-lived format (that's not email). So with that, I have > started > https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking > > (I figured there's still more discussion to be had before it becomes a SIP > with > technical requirements). > >>>> > >>>> I encourage anyone and everyone who is interested to provide their input > (comment or edit). This is a community initiative, and shouldn't be limited > by > me or any biases I may have. Hopefully people find this useful in moving the > discussion forward. > >>>> > >>>> -Kevin > >>> > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
