Here is that drop: https://github.com/markrmiller/solr/tree/ref-branch
It's broken up into two commits: the original work and the newer work. The original work consisted of thousands of random checkpoint commits with no rhyme or reason, so I consolidated it into one commit. I also used AI assistance to enhance the comments, making the code more navigable. The original commit has an example of an extremely scalable and pretty simple Overseer implementation. The second commit takes that to another level. On Sat, Oct 31, 2026 at 2:01 PM Mark Miller <[email protected]> wrote: > > Interestingly, the JVM is not really the limiting factor here. > > On Luxir's full text search and string facet benchmarks, Solr can outperform > Luxir with a geometric mean of almost 3x (at 1, 8, and 32 concurrency) > without using any native code. > > That largely just takes better faceting code and a bunch of faster query / > weight / scorer classes. > > I'm stepping away from Open Source it looks, but before I do, I'll drop the > reference branch code from 5 years ago that was never shared and some AI > accelerated work on top of that that demonstrates that performance and > removes any dependency on an external system, like Zookeeper, among a slew of > other things - including some native acceleration, but none of it needed for > those benchmark wins. > > Feel free to steal ideas / code / or whatever from it. > > - MRM > > On Wed, Oct 28, 2026 at 8:56 AM David Smiley <[email protected]> wrote: >> >> Definitely cool Ishan & team! I think it's super clever to do this at a >> Lucene codec level. I'm surprised Luxir is able to deliver impressive >> gains in this way in spite of "only" being used as a codec. >> I also saw Yonik's Luxir presentation on Monday. >> >> Do you plan on contributing the Solr patches by themselves, assuming they >> are each appropriate? I didn't look at them in detail. Your descriptions >> suggested utility without Luxir. >> >> On Fri, Oct 23, 2026 at 3:23 PM Ishan Chattopadhyaya < >> [email protected]> wrote: >> >> > Hi all, >> > >> > For years, Solr has been only as fast as Lucene allows. Every gain in >> > storage formats, query execution or faceting has had to come from upstream, >> > on Lucene's timeline and within the limits of the JVM. At SearchScale, we >> > wanted to see how far Solr could go without waiting on that, so we've >> > started exploring what it would take to swap Lucene's internals for >> > something faster: Luxir <https://github.com/luxir-search/luxir/>, a C++ >> > search engine, by Yonik Seeley! >> > >> > SoLux runs Luxir at the codec level, and the split works like this: >> > >> > - *Solr and Lucene still handle:* documents, analysis, segments, >> > commits, NRT and SolrCloud. >> > - *Luxir takes over:* storing the postings, doc values, points and >> > stored fields, and running queries, filters, facets and joins natively >> > in >> > one call per request, through the FFM API, with no JNI. >> > - *Packaging:* it ships as a Solr module (luxir-codec), enabled through >> > the configset. >> > >> > *Headline numbers.* We indexed 100M e-commerce events into 3 nodes × 3 >> > shards, then ran four query types from 16 concurrent clients, on a Ryzen 9 >> > 7950X and a Ryzen 9 5950X: >> > >> > SoLux vs stock Solr >> > Indexing 1.8–2.1× faster >> > Terms facet throughput 2.0–2.1× >> > Range facet throughput 2.7–2.8× >> > Join throughput 5.9–6.0× >> > All query tasks, total time 2.1× less >> > product_id terms facet + avg(price) 0.55–0.57×, slower: our next target >> > >> > The benchmark, full results, exact versions and step-by-step instructions >> > to reproduce it are in luxir.md. It also lists the changes behind these >> > numbers and what each one contributed: >> > https://github.com/SearchScale/solr-bench/blob/master/luxir.md >> > >> > *Changes to Luxir.* >> > >> > SoLux runs Luxir with four patches on top of upstream Luxir project, kept >> > in luxir-lucene. These are needed to improve on query and indexing >> > performance. >> > >> > - 0001 >> > < >> > https://github.com/SearchScale/luxir-lucene/blob/main/luxir-patches/0001-query-materialize-large-point-ranges-as-bitsets.patch >> > >: >> > build large point ranges as bitsets instead of sorting doc arrays. >> > - 0002 >> > < >> > https://github.com/SearchScale/luxir-lucene/blob/main/luxir-patches/0002-query-skip-term-windows-without-docs-when-counting.patch >> > >: >> > when counting, jump to the window holding a term's next doc instead of >> > stepping through every window. >> > - 0003 >> > < >> > https://github.com/SearchScale/luxir-lucene/blob/main/luxir-patches/0003-reader-seek-explicit-term-sets-block-by-block.patch >> > >: >> > look up explicit term sets block by block, decoding only the requested >> > terms' metadata. >> > - 0004 >> > < >> > https://github.com/SearchScale/luxir-lucene/blob/main/luxir-patches/0004-query-answer-sparse-range-verification-from-block-re.patch >> > >: >> > answer a range check behind a selective filter from each column block's >> > min >> > and max, reading values only in blocks that cross the range. Together >> > with >> > a faster range-facet bucket lookup, this took filtered range facets from >> > 14.2 to 5.3 ms per shard. >> > >> > *Caveats:* >> > >> > - This is experimental. Each configuration ran once. >> > - The Luxir nodes run with the security manager off, so the native >> > library can load. >> > - SoLux still sits on Lucene's segments and needs four small Solr core >> > changes. >> > >> > I'd love your feedback, and results from your own hardware. >> > >> > Thanks, >> > Ishan >> > -- - MRM --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
