Hello All,
Following up on David's "State of Lucene Index Sorting" report from a few
weeks back. That report reads as a roadmap — a handful of related tickets
that together would make index sorting first-class in Solr — so I spent
time working through them end to end. I have local, tested implementations
for the open pieces and I'd like to bring them to the list before opening
PRs, since several sit in areas others own (particularly Christine's #313).
I'll lay out how I see the puzzle, what I've built, and where I'd like
direction / to collaborate.
*The picture (from David's report)*
Index sorting works today, but only through the indirect
`SortingMergePolicyFactory` — a merge policy that does no merging and
exists only to carry a `Sort` to `IndexWriterConfig.setIndexSort()`.
The report flagged five gaps; working through them surfaced two adjacent
pieces (SOLR-15390, SOLR-17310).
The full map as I now see it:
- SOLR-13681 — direct <indexSort> config — Open, Christine's draft #313
- SOLR-9108 — improve sort config (overlaps 13681) — Open
- SOLR-12230 — deprecate SortingMergePolicy — Open
- SOLR-12239 — re-sort an existing collection — Open, my PR #4644
- SOLR-17170 — nested/block-join sorting — Resolved (David)
- SOLR-15390 — modernize the early-termination collector (adjacent) — Patch
Available
- SOLR-17310 — configurable leaf sorter (adjacent) — Open, Wei Wang's #2477
(stale-bot auto-closed, not rejected)
*What I've implemented and verified locally*
1. SOLR-13681 — direct <indexSort> config.
A first-class <indexConfig><indexSort>timestamp
desc</indexSort></indexConfig>, parsed with the same SortSpecParsing the
query sort uses, replacing the merge-policy indirection. This reimplements
Christine's #313 approach on current main (the draft predates `MapWriter`
and David's SOLR-17170 parent-field work, so it needed reworking) and
composes with the nested-docs parent field. Christine — this is your lane
and I'm not looking to step on #313. I'd rather hand this over or work with
you on it — or simply absorb any of it into #313 as you see fit, with or
without me. I just found it was the keystone the other pieces depend on,
and its long-standing blocker ("what about existing collections?") is
answered by SOLR-12239.
2. SOLR-12239 — re-sort an existing collection (PR #4644, already up).
A RESORTINDEX core-admin action (v1 + v2) using LUCENE-9484
(SortingCodecReader + addIndexes) to sort pre-existing segments without
a full reindex — removing the "sort is immutable, must reindex" limitation.
It reads the configured <indexSort> as its target, so the workflow is: set
<indexSort> → run RESORTINDEX once.
3. SOLR-17310 — configurable leaf sorter (<segmentSort>).
Builds on Wei Wang's #2477 (credited). Exposes Lucene's `setLeafSorter` for
*between-segment* ordering, distinct from index sort's *within-segment*
order. Generalized beyond the time-only version to also accept a
numeric-field spec, since setLeafSorter takes any Comparator<LeafReader>.
4. SOLR-15390 — modernize the deprecated collector.
Solr's segmentTerminateEarly rides a
@Deprecated EarlyTerminatingSortingCollector; I swapped it for Lucene's
native TopFieldCollector.isEarlyTerminated(), preserving the
`segmentTerminatedEarly` response semantics. The ticket's broader ask is
deprecating the segmentTerminateEarly param in favor of `minExactCount` — I
did the internal modernization but left the param-deprecation as a separate
decision, since it's user-facing.
5. SOLR-12230 — deprecate SortingMergePolicy(Factory).
Once <indexSort> exists as the replacement, the old carrier can be
deprecated. Done as the last step, with a Solr-11 upgrade note.
Everything is tested (unit + SolrCloud; ~40 tests) and passes errorprone /
check / ref-guide link checks.
*Two design questions I'd like the list's view on*
- Config shape: <indexSort> (within-segment) and <segmentSort>
(between-segment) as *separate* <indexConfig> elements, vs. the
<indexSorters> unified-element idea from SOLR-17310. My reading
of precedent leans separate — Lucene keeps `setIndexSort`/`setLeafSorter`
as distinct setters, and ES/OpenSearch expose only the within-segment axis
(index.sort.*) with no leaf-sorter equivalent — but I don't feel strongly
and want to hear opinions.
- segmentTerminateEarly vs minExactCount: actually deprecate the param
(SOLR-15390's literal ask), or just modernize the internals as I've done?
What I'm asking is mainly direction and collaboration — especially with
Christine on 13681, since that's the keystone and her active work. If
there's interest, I'll open the pieces as separate per-ticket PRs (#4644
is already up). OR open PR with all changes to see full picture. Happy to
adjust any of it.
Thanks,
Serhiy
On Wed, 15 Jul 2026 at 20:01, David Smiley <[email protected]> wrote:
> I'm working on a project / idea that will require it. I'll share more
> about that later when appropriate, but it's too early now.
>
> At least the current state isn't bad. It works, especially with expanded
> functionality extending to schemas/use-cases with nested docs: SOLR-17170
> -- fully merged back to 9.x. There's a usability issue that I hope
> Christine will prioritize improving.
>
> I've heard Adrien and others extol the virtues/benefits of index sorting
> but haven't yet had deployed it to share. I will when I can.
>
> On Wed, Jul 15, 2026 at 8:06 AM Jason Gerlowski <[email protected]>
> wrote:
>
> > Curious David - are you considering using index sorting for a
> > use-case, or have any experience with it to share in terms of
> > speedups, etc? Or more just something that struck your fancy?
> >
> > I remember index-sorting being something that came up in a Community
> > over Code "Birds of a Feather" session (in, I believe, Halifax?), as
> > one of the things Lucene supported that Solr should really consider
> > enabling out of the box for performance benefits. But I can't
> > remember the details of that discussion very well...
> >
> > Best,
> >
> > Jason
> >
> > On Thu, Jul 2, 2026 at 12:20 PM David Smiley <[email protected]> wrote:
> > >
> > > I started investigating the current state of Lucene's index sorting
> > support
> > > in Solr. I had Claude Opus write a report for me. Rather than hoard
> it
> > to
> > > myself, I'm sharing with everyone in case others are wondering what's
> up
> > as
> > > well.
> > >
> > >
> > > Background
> > > ----------
> > >
> > > Lucene has supported index-level sorting since LUCENE-6766 (Lucene
> 6.2),
> > > where segments are internally sorted by a configurable field order at
> > > flush/merge time. This enables significant query-time optimizations --
> > > when the query's sort matches the index sort, Lucene can skip entire
> > > segments or terminate collection early.
> > >
> > >
> > > Current Solr Support
> > > --------------------
> > >
> > > Solr does support index sorting today, but through an indirect
> mechanism:
> > >
> > > Configuration is done via SortingMergePolicyFactory in solrconfig.xml:
> > >
> > > <mergePolicyFactory
> > > class="org.apache.solr.index.SortingMergePolicyFactory">
> > > <str name="sort">timestamp desc</str>
> > > <str name="wrapped.prefix">inner</str>
> > > <str
> > > name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str>
> > > </mergePolicyFactory>
> > >
> > > Internally, SortingMergePolicy is a FilterMergePolicy that does nothing
> > > merge-policy-related -- it simply holds a Sort object. SolrIndexConfig
> > > then has a special instanceof check that extracts this Sort and calls
> > > IndexWriterConfig.setIndexSort(). The class itself has a TODO comment
> > > acknowledging this is a workaround: "remove this and add indexSort
> > > specification directly to solrconfig.xml?"
> > >
> > > Query-side integration exists via the "segmentTerminateEarly" query
> > > parameter, which wraps the collector in an
> > EarlyTerminatingSortingCollector.
> > > Note that this collector is @Deprecated -- modern Lucene's
> > TopFieldCollector
> > > handles early termination natively when it detects sorted segments.
> > >
> > > The /admin/segments API (with coreInfo=true) exposes the indexSort
> > > configuration and per-segment sort info.
> > >
> > > AtomicUpdateDocumentMerger correctly detects fields used for index
> > sorting
> > > and prevents DocValues-only updates on them (a Lucene limitation).
> > >
> > >
> > > Open Issues
> > > -----------
> > >
> > > Several open JIRA issues relate to this area.
> > >
> > > SOLR-9108: Improve how index time sorting is configured
> > > https://issues.apache.org/jira/browse/SOLR-9108
> > >
> > > Filed by Mike McCandless in 2016 right after LUCENE-6766. Proposes
> > > configuring index sort directly in solrconfig.xml alongside other
> > > IndexWriter settings rather than piggybacking on the merge policy.
> > >
> > > SOLR-13681: Make Lucene's index sorting directly configurable in Solr
> > > https://issues.apache.org/jira/browse/SOLR-13681
> > >
> > > Filed by Christine Poerschke in 2019. Has a draft PR (#313) that adds
> > > a direct <indexSort> config element to solrconfig.xml. The PR has
> been
> > > stalled since 2021; the main open question is what should happen when
> > > index sorting is enabled on an existing collection that already has
> > > unsorted segments. Duplicates SOLR-12230 (deprecate
> > SortingMergePolicy).
> > >
> > > SOLR-12239: Enabling index sorting causes CorruptIndexException
> > > https://issues.apache.org/jira/browse/SOLR-12239
> > >
> > > When index sorting is enabled on an existing collection with unsorted
> > > segments, reloading throws: "segment not sorted with indexSort=null".
> > > The current workaround is to delete all data and reindex from
> scratch.
> > > Notably, the related LUCENE-9484 ("Allow index sorting to happen
> after
> > > the fact") was fixed in Lucene 9.0, which allows merging unsorted
> > > segments into sorted ones retroactively. Solr has not wired this up.
> > >
> > > SOLR-17170: Support Blocks in Index Sorting
> > > https://issues.apache.org/jira/browse/SOLR-17170
> > >
> > > Lucene 9.10+ supports block-aware presort during index sorting (via
> > > Lucene PR #12829). This is critical for nested/block-join documents.
> > > No Solr-side work has been done.
> > >
> > >
> > > Summary
> > > -------
> > >
> > > Index sorting in Solr works for simple (non-nested) use cases via the
> > > SortingMergePolicyFactory, but the implementation is showing its age:
> > >
> > > - Configuration is indirect and hacky (merge policy as a Sort carrier)
> > > - Cannot be safely enabled on existing collections without full
> reindex,
> > > despite Lucene having solved this at the engine level since 9.0
> > > - Incompatible with nested/block-join documents on Solr 10+
> > > - The query-side early termination collector is deprecated
> > > - A draft PR for direct configuration has been stalled since 2021
> > >
> > > ~ David Smiley
> > > Apache Lucene/Solr Search Developer
> > > http://www.linkedin.com/in/davidwsmiley
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
> >
>
