I want to share with the community that earlier this week I had the opportunity to visit Kevin and team and talk about Solr Orbit fitting thier specific use case, and I think we have consensus that this solution is how we want to move forward. I know it will solve my client's needs as well.
I checked in with Jan, and I think the next step is to get a vote thread on bringing in this codebase as a `solr-orbit` subproject. On 2026/04/23 20:42:30 Eric Pugh wrote: > Home after some travel... > > I think the solr-benchmark-game needs a bit more marinating to see the true > value. Also, Kevin and (???) did some great analysis on it and pointed out > some big weaknesses to it's approach. > > I'm onboard with the solr-orbit plan, and can see how I would use it with my > client to establish the benchmarks needed. > > On 2026/04/17 20:38:17 "Kevin Liang (BLOOMBERG/ 919 3RD A) via dev" wrote: > > "Solr Orbit" seems a fine name to me. > > > > Re: David's point on picking one of search-benchmark-game vs. > > solr-benchmark-fork > > > > I'll add my opinion as I've used / made code changes to both and have > > provided this feedback to Eric / Jan. > > > > The search-benchmark-game is nice for its simplicity and results > > presentation. It can quickly give you some data points between two versions > > of software. That being said, at best it seems geared more towards > > assessing a library (lucene) than a full distributed system (solr). Its > > feature set is limited and the methodology for executing queries / drawing > > statistical conclusions leaves something to be desired (can expand in > > detail if anyone wants to hear). > > > > In contrast, solr-benchmark (OSB fork) is building off a feature set > > constructed over the better part of a decade. It's a much larger codebase > > and a more mature benchmarking framework with much greater capability to do > > different kinds of performance test workloads. > > > > Personally I would go with solr-benchmark. Building a web interface on top > > (or hooking up a jenkins build pipeline) can be done. And like we said in > > the meeting, just getting people to run the tool independently and > > publishing their results, even as plain text on the wiki, would be a good > > first step win. > > > > From: [email protected] At: 04/16/26 13:00:14 UTC-4:00To: > > [email protected] > > Subject: Re: Solr Performance Benchmarking Discussion Page > > > > It's not work using Elastic Inc's trademark "Rally" if we can avoid it. > > They > > are known to pursue such violations. And I believe it is a violation of the > > Apache license too. I like "Solr Orbit" :) > > > > I'll wait some time to let all voices be heard wrt the various tools and > > which > > to pick going forward, then if everyone rallies (pun intended) around > > solr-benchmark, then we can proceed. > > > > Jan > > > > > 16. apr. 2026 kl. 14:36 skrev David Smiley <[email protected]>: > > > > > > Why avoid the word "Rally" if that's its foundation/lineage? If we > > > want to avoid risk of using that word, then we simply ask > > > ElasticSearch with our proposed name for our fork that includes that > > > word. > > > > > > On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <[email protected]> wrote: > > >> > > >> Hi, > > >> > > >> Thanks for presenting the solr-benchmark tool on the community meetup > > yesterday Kevin. > > >> I'll encourage others to take it for a spin. There are rough edges but > > >> you > > should be able to get some results. > > >> > > >> We also discussed the potential for moving the tool from my github space > > >> to > > asf. > > >> That would make it easier for the community to contribute and > > >> collaborate. > > But it requires 2-3 PMC members interested in being maintainers. > > >> If we make it an asf repo, David noted that we are not required to > > >> publish > > official releases until we feel compelled to do so. > > >> Also, David noted the naming collision with the internal solr-benchmark > > module. How about "solr-orbit", which is a homage to "Rally" (orbiting the > > race > > track) but in our Solar-system universe :) > > >> > > >> As a next step I'll start a VOTE thread for accepting the two repos into > > ASF. The results of the VOTE will guide further action. > > >> > > >> Jan > > >> > > >>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <[email protected]>: > > >>> > > >>> Hi, > > >>> > > >>> I read through the benchmarking wiki page today. It was well written, > > thanks Kevin for reviving this effort, and in such a structured way! > > >>> > > >>> There's a renewed energy around benchmarking, and more tooling options > > >>> than > > ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr > > Benchmark": > > >>> > > >>> https://github.com/janhoy/solr-benchmark > > >>> > > >>> You may quickly notice that this looks familiar. And yes, it is a > > >>> fork/port > > of Rally/OpensearchBenchmark, ported to provision and benchmark Solr > > clusters, > > using the same datasets and "workload"s as those tools. > > >>> I first tried to start a fork 2 or 3 years ago but it stranded. I made > > >>> a > > new effort using LLM agents and this first working version was prepared in > > a > > few afternoons. > > >>> Even if the foundation is solid and proven over many years, this > > >>> initial > > port is not complete, view it as a MVP and WIP. Only one workload / dataset > > is > > ported so far. Take it for a spin... > > >>> > > >>> I'll defer to Kevin to add it to the tools list of the wiki and > > >>> continue > > the analysis effort with this as one of the contenders. > > >>> > > >>> Jan > > >>> > > >>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A) > > <[email protected]>: > > >>>> > > >>>> Hello everyone, > > >>>> > > >>>> Given the recent interest and discussion around Solr performance > > benchmarking, I figured it would be useful to 1) centralize the discussion > > and > > 2) bring it to a long-lived format (that's not email). So with that, I have > > started > > https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking > > > > (I figured there's still more discussion to be had before it becomes a SIP > > with > > technical requirements). > > >>>> > > >>>> I encourage anyone and everyone who is interested to provide their > > >>>> input > > (comment or edit). This is a community initiative, and shouldn't be limited > > by > > me or any biases I may have. Hopefully people find this useful in moving > > the > > discussion forward. > > >>>> > > >>>> -Kevin > > >>> > > >> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
