[
https://issues.apache.org/jira/browse/SOLR-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099684#comment-18099684
]
Serhiy Bzhezytskyy commented on SOLR-12239:
-------------------------------------------
Thanks David for this feedback.
> In the days of AI code gen, the limited resources is code review time (my
> time)
Support you 1000%.
> There's understandably a lot of code for this complicated feature.
True, and especially for a newcomer as me.
> I suspect you implemented this simply because you saw my email and this
> issue, not because you actually want to put this to use in your use of Solr
Exactly :) This issue and PR were my first attempt to dive into this area.
So yes — I did this PR, and then realised I wanted to understand the whole
puzzle rather than one ticket of it, which became my second
attempt:[#4648|https://github.com/apache/solr/pull/4648], the all-in-one across
SOLR-15390/13681/12239/17310/12230.
Now the technical details:
*1. Updates during the resort are lost — your suspicion was right.*
ResortCoreIndex opens a DirectoryReader on the current index, writes the
re-sorted copy, then repoints the core via `modifyIndexProps`, and nothing
marks the core read-only across that window. Updates arriving mid-resort go to
the old writer and are dropped at the swap. I see you had this same read-only
discussion with Rahul on SOLR-18190 in April; my PR doesn't even name the
problem.
*2. The two-step config hazard is real, and my own code documents it as the
design.* The comment in ResortCoreIndex describes the workflow as "set
<indexSort>, then RESORTINDEX" — so there's a window where solrconfig declares
a sort the segments don't have.
*3. The primitive underneath is experimental and memory-hungry.*
SortingCodecReader is @lucene.experimental and its javadoc says it "should only
be used for merging" because "pulling fields from this reader might be very
costly and memory intensive" — it allocates maxDoc-proportional heap per
docValues field. LUCENE-9539, opened by the author of LUCENE-9484 nineteen days
after he shipped it, says the class needs "multiple steps towards a more
useable version". Open since 2020.
*4. No other engine exposes this,* which answers your "is that too much to ask
of a user?" — index.sort.* is final in both OpenSearch and Elasticsearch, and
the ES docs say plainly that adding or updating a sort on an existing index is
not allowed. Both answer reindex. So you concern is valid.
*5. What is worth keeping* — Christine asked in 2019 and 2020 whether
LUCENE-9484 solves this, and it does: SortingCodecReader.wrap per leaf plus
addIndexes into a fresh sort-configured writer, first released in 8.7.0. One
trap for whoever wires it up — a core with a configured SortingMergePolicy
carries _is_root_ on its source segments, so a resort writer that doesn't
declare a parent field makes addIndexes refuse to merge.
*Action item I'd suggest:* I close #4644, and we leave this JIRA open — the
underlying problem is still real, and what's on it now is the answer to
Christine's question rather than a proposal to merge. If you'd rather resolve
it as Won't Fix, that's your call and I won't argue.
> Enabling index sorting causes "segment not sorted with indexSort=null"
> ----------------------------------------------------------------------
>
> Key: SOLR-12239
> URL: https://issues.apache.org/jira/browse/SOLR-12239
> Project: Solr
> Issue Type: Improvement
> Affects Versions: 7.1
> Reporter: Ishan Chattopadhyaya
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When index sorting is enabled on an existing collection/index (using
> SortingMergePolicy), the collection reload causes the following exception:
> {code}
> java.util.concurrent.ExecutionException:
> org.apache.solr.common.SolrException: Unable to create core
> [mycoll_shard1_replica_n1]
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.core.CoreContainer.lambda$load$14(CoreContainer.java:671)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.solr.common.SolrException: Unable to create core
> [mycoll_shard1_replica_n1]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1045)
> at
> org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:642)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
> ... 5 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:989)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:844)
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1029)
> ... 7 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2076)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2196)
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1072)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:961)
> ... 9 more
> Caused by: org.apache.lucene.index.CorruptIndexException: segment not sorted
> with indexSort=null (resource=_0(7.1.0):C1)
> at
> org.apache.lucene.index.IndexWriter.validateIndexSort(IndexWriter.java:1185)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108)
> at
> org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:119)
> at
> org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:94)
> at
> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
> at
> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:131)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2037)
> ... 12 more
> {code}
> This means that the user actually needs to delete the index segments, reload
> the collection and then re-index. This is bad user experience.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
