[
https://issues.apache.org/jira/browse/LUCENE-8264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451952#comment-16451952
]
Simon Willnauer commented on LUCENE-8264:
-----------------------------------------
I totally agree with robert here, good collection of valid technical points. We
can't let lurking corruptions happen. The improvements made to norms here are
awesome and we need to move forward with stuff like this. Also after looking at
the details, I am convinced the guarantees that this restriction gives us are
crucial to the future of lucene. We can't support lurking corruptions for users
that come from ancient versions by converting (merging / rewriting segments)
from N-X to N in steps that nobody ever tested.
Also the points about the database aspect are very much valid. We need raw data
to re-create these indices reliably and if you are running on top of a search
engine you need to account for reindexing.
Btw. we have this restriction in ES since 1.0 implicitly. We always only
supported N-1 major versions for ES indices, yet they happen to be
corresponding to N-1 Lucene major versions. There is also a lot of work gone
into supporting searching across major versions of ES to allow users to stay on
older versions for retention policy purposes. Some of these conversations are
not easy but necessary for us to prevent support insanity.
That said, I think there might be room for N-X at some point as long as the
guarantee is only N-1. At some point we might allow the min index created
version to be 7 even if you are on 9. But for us to make progress we need to be
free to break and only guarantee N-1.
Also, what this means is that your indices are supported ~2.5 years that is the
major release cadence historically. I think it's important to keep this in
mind.
> Allow an option to rewrite all segments
> ---------------------------------------
>
> Key: LUCENE-8264
> URL: https://issues.apache.org/jira/browse/LUCENE-8264
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Erick Erickson
> Assignee: Erick Erickson
> Priority: Major
>
> For the background, see SOLR-12259.
> There are several use-cases that would be much easier, especially during
> upgrades, if we could specify that all segments get rewritten.
> One example: Upgrading 5x->6x->7x. When segments are merged, they're
> rewritten into the current format. However, there's no guarantee that a
> particular segment _ever_ gets merged so the 6x-7x upgrade won't necessarily
> be successful.
> How many merge policies support this is an open question. I propose to start
> with TMP and raise other JIRAs as necessary for other merge policies.
> So far the usual response has been "re-index from scratch", but that's
> increasingly difficult as systems get larger.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
