[jira] [Updated] (SOLR-18096) Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade
[
https://issues.apache.org/jira/browse/SOLR-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Goswami updated SOLR-18096:
-
Description:
Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The
merge policy (LatestVersionMergePolicy) part of the overall enhancement made it
into Solr 10 as part of SOLR-17725 which enables users to be able to upgrade
their index for the next Solr version (Solr 11), although with more write
overhead and manual steps.
The other part of the enhancement is this JIRA which proposes exposing an
UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to
achieve the upgrade in a highly targeted and efficient way by only upgrading
older segments and abstracts the setting/unsetting of the merge policy.
{+}*Design/Operational flow*{+}:
The UPGRADECOREINDEX CoreAdmin command rewrites segments written by older
Lucene versions into the current format (as long as the fields are stored=true
or docValues=true). This makes it possible to use the same index across
multiple major versions without requiring reindexing from the original data
source .
For each core, it:
1. Opens the existing index and sets
[LatestVersionMergePolicy|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/index/LatestVersionMergePolicy.java]
on the IndexWriter to prevent older-format segments from merging with
latest-format segments.
2. Identifies segments written by an older Lucene major version
([shouldUpgradeSegment()|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java#L250]
– any segment whose minVersion predates the current major version).
3. Reads every document from those old-format segments, reconstructing
SolrInputDocuments with all stored/docValues fields.
4. Re-adds each document through an update processor chain, which writes it
as a new current-format segment.
5. Commit removes the now-obsolete old segments since they don't contain any
live docs.
6. Restores the original merge policy.
Documents that already reside in current-format segments are untouched. The
operation is idempotent – re-running it skips segments that are already up to
date.
was:
Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The
merge policy (LatestVersionMergePolicy) part of the overall enhancement made it
into Solr 10 as part of SOLR-17725 which enables users to be able to upgrade
their index for the next Solr version (Solr 11), although with more write
overhead and manual steps.
The other part of the enhancement is this JIRA which proposes exposing an
UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to
achieve the upgrade in a highly targeted and efficient way by only upgrading
older segments and abstracts the setting/unsetting of the merge policy.
Solr's UPGRADECOREINDEX CoreAdmin command rewrites segments written by older
Lucene versions into the current format (as long as the fields are stored=true
or docValues=true). This makes it possible to use the same index across
multiple major versions without requiring reindexing from the original data
source .
For each core, it:
1. Opens the existing index and sets
[LatestVersionMergePolicy|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/index/LatestVersionMergePolicy.java]
on the IndexWriter to prevent older-format segments from merging with
latest-format segments.
2. Identifies segments written by an older Lucene major version
([shouldUpgradeSegment()|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java#L250]
– any segment whose minVersion predates the current major version).
3. Reads every document from those old-format segments, reconstructing
SolrInputDocuments with all stored fields.
4. Re-adds each document through an update processor chain, which writes it
as a new current-format segment.
5. Commits and runs expungeDeletes to remove the now-obsolete old segments.
Also restores the original merge policy.
Documents that already reside in current-format segments are untouched. The
operation is idempotent – re-running it skips segments that are already up to
date.
> Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade
> --
>
> Key: SOLR-18096
> URL: https://issues.apache.org/jira/browse/SOLR-18096
> Project: Solr
> Issue Type: Improvement
>Reporter: Rahul Goswami
>Priority: Major
> Labels: pull-request-available
> Fix For: 10.1, 9.11
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion.
> The merge
[jira] [Updated] (SOLR-18096) Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade
[ https://issues.apache.org/jira/browse/SOLR-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Goswami updated SOLR-18096: - Description: Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 as part of SOLR-17725 which enables users to be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead and manual steps. The other part of the enhancement is this JIRA which proposes exposing an UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracts the setting/unsetting of the merge policy. Solr's UPGRADECOREINDEX CoreAdmin command rewrites segments written by older Lucene versions into the current format (as long as the fields are stored=true or docValues=true). This makes it possible to use the same index across multiple major versions without requiring reindexing from the original data source . For each core, it: 1. Opens the existing index and sets [LatestVersionMergePolicy|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/index/LatestVersionMergePolicy.java] on the IndexWriter to prevent older-format segments from merging with latest-format segments. 2. Identifies segments written by an older Lucene major version ([shouldUpgradeSegment()|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java#L250] – any segment whose minVersion predates the current major version). 3. Reads every document from those old-format segments, reconstructing SolrInputDocuments with all stored fields. 4. Re-adds each document through an update processor chain, which writes it as a new current-format segment. 5. Commits and runs expungeDeletes to remove the now-obsolete old segments. Also restores the original merge policy. Documents that already reside in current-format segments are untouched. The operation is idempotent – re-running it skips segments that are already up to date. was: Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 as part of SOLR-17725 which enables users to be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead and manual steps. The other part of the enhancement is this JIRA which proposes exposing an UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracts the setting/unsetting of the merge policy. > Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade > -- > > Key: SOLR-18096 > URL: https://issues.apache.org/jira/browse/SOLR-18096 > Project: Solr > Issue Type: Improvement >Reporter: Rahul Goswami >Priority: Major > Labels: pull-request-available > Fix For: 10.1, 9.11 > > Time Spent: 1h > Remaining Estimate: 0h > > Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. > The merge policy (LatestVersionMergePolicy) part of the overall enhancement > made it into Solr 10 as part of SOLR-17725 which enables users to be able to > upgrade their index for the next Solr version (Solr 11), although with more > write overhead and manual steps. > The other part of the enhancement is this JIRA which proposes exposing an > UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to > achieve the upgrade in a highly targeted and efficient way by only upgrading > older segments and abstracts the setting/unsetting of the merge policy. > Solr's UPGRADECOREINDEX CoreAdmin command rewrites segments written by older > Lucene versions into the current format (as long as the fields are > stored=true or docValues=true). This makes it possible to use the same index > across multiple major versions without requiring reindexing from the original > data source . > For each core, it: > 1. Opens the existing index and sets > [LatestVersionMergePolicy|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/index/LatestVersionMergePolicy.java] > on the IndexWriter to prevent older-format segments from merging with > latest-format segments. > 2. Identifies segments written by an older Lucene major version > ([shouldUpgradeSegment()|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java#L250] > – any segment whose minVersion predates the cur
[jira] [Updated] (SOLR-18096) Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade
[ https://issues.apache.org/jira/browse/SOLR-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ricardo Anguiano updated SOLR-18096: Reporter: Rahul Goswami (was: Rahul Goswami) > Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade > -- > > Key: SOLR-18096 > URL: https://issues.apache.org/jira/browse/SOLR-18096 > Project: Solr > Issue Type: Improvement >Reporter: Rahul Goswami >Priority: Major > Labels: pull-request-available > Fix For: 10.1, 9.11 > > Time Spent: 1h > Remaining Estimate: 0h > > Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. > The merge policy (LatestVersionMergePolicy) part of the overall enhancement > made it into Solr 10 as part of SOLR-17725 which enables users to be able to > upgrade their index for the next Solr version (Solr 11), although with more > write overhead and manual steps. > The other part of the enhancement is this JIRA which proposes exposing an > UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to > achieve the upgrade in a highly targeted and efficient way by only upgrading > older segments and abstracts the setting/unsetting of the merge policy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (SOLR-18096) Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade
[ https://issues.apache.org/jira/browse/SOLR-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Goswami updated SOLR-18096: - Description: Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 as part of SOLR-17725 which enables users to be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead and manual steps. The other part of the enhancement is this JIRA which proposes exposing an UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracts the setting/unsetting of the merge policy. was: Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 as part of SOLR-17725 which enables users to be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead and manual steps. The other part of the enhancement is this JIRA which proposes exposing an UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracts the setting of the merge policy. > Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade > -- > > Key: SOLR-18096 > URL: https://issues.apache.org/jira/browse/SOLR-18096 > Project: Solr > Issue Type: Improvement >Reporter: Rahul Goswami >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. > The merge policy (LatestVersionMergePolicy) part of the overall enhancement > made it into Solr 10 as part of SOLR-17725 which enables users to be able to > upgrade their index for the next Solr version (Solr 11), although with more > write overhead and manual steps. > The other part of the enhancement is this JIRA which proposes exposing an > UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to > achieve the upgrade in a highly targeted and efficient way by only upgrading > older segments and abstracts the setting/unsetting of the merge policy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (SOLR-18096) Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade
[ https://issues.apache.org/jira/browse/SOLR-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Goswami updated SOLR-18096: - Description: Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 as part of SOLR-17725 which enables users to be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead and manual steps. The other part of the enhancement is this JIRA which proposes exposing an UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracts the setting of the merge policy. was: Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 as part of SOLR-17725 which enables users to be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead and manual steps. The other part of the enhancement is this JIRA which proposes exposing an UPRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracts the setting of the merge policy. > Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade > -- > > Key: SOLR-18096 > URL: https://issues.apache.org/jira/browse/SOLR-18096 > Project: Solr > Issue Type: Improvement >Reporter: Rahul Goswami >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. > The merge policy (LatestVersionMergePolicy) part of the overall enhancement > made it into Solr 10 as part of SOLR-17725 which enables users to be able to > upgrade their index for the next Solr version (Solr 11), although with more > write overhead and manual steps. > The other part of the enhancement is this JIRA which proposes exposing an > UPGRADECOREINDEX CoreAdmin API for standalone installations to be able to > achieve the upgrade in a highly targeted and efficient way by only upgrading > older segments and abstracts the setting of the merge policy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (SOLR-18096) Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade
[ https://issues.apache.org/jira/browse/SOLR-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Goswami updated SOLR-18096: - Description: Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 as part of SOLR-17725 which enables users to be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead and manual steps. The other part of the enhancement is this JIRA which proposes exposing an UPRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracts the setting of the merge policy. was: Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 which enables uses to still be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead. The other part of the enhancement is this JIRA which proposes exposing an UPRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracting the setting of the merge policy. > Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade > -- > > Key: SOLR-18096 > URL: https://issues.apache.org/jira/browse/SOLR-18096 > Project: Solr > Issue Type: Improvement >Reporter: Rahul Goswami >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. > The merge policy (LatestVersionMergePolicy) part of the overall enhancement > made it into Solr 10 as part of SOLR-17725 which enables users to be able to > upgrade their index for the next Solr version (Solr 11), although with more > write overhead and manual steps. > The other part of the enhancement is this JIRA which proposes exposing an > UPRADECOREINDEX CoreAdmin API for standalone installations to be able to > achieve the upgrade in a highly targeted and efficient way by only upgrading > older segments and abstracts the setting of the merge policy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (SOLR-18096) Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade
[ https://issues.apache.org/jira/browse/SOLR-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SOLR-18096: -- Labels: pull-request-available (was: ) > Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade > -- > > Key: SOLR-18096 > URL: https://issues.apache.org/jira/browse/SOLR-18096 > Project: Solr > Issue Type: Improvement >Reporter: Rahul Goswami >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. > The merge policy (LatestVersionMergePolicy) part of the overall enhancement > made it into Solr 10 which enables uses to still be able to upgrade their > index for the next Solr version (Solr 11), although with more write overhead. > The other part of the enhancement is this JIRA which proposes exposing an > UPRADECOREINDEX CoreAdmin API for standalone installations to be able to > achieve the upgrade in a highly targeted and efficient way by only upgrading > older segments and abstracting the setting of the merge policy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (SOLR-18096) Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade
[ https://issues.apache.org/jira/browse/SOLR-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Goswami updated SOLR-18096: - Description: Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 which enables uses to still be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead. The other part of the enhancement is this JIRA which proposes exposing an UPRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracting the setting of the merge policy. was: Forking this JIRA from parent SOLR-1775 based on [~dsmiley]'s suggestion. The merge policy (LatestVersionMergePolicy) part of the overall enhancement made it into Solr 10 which enables uses to still be able to upgrade their index for the next Solr version (Solr 11), although with more write overhead. The other part of the enhancement is this JIRA which proposes exposing an UPRADECOREINDEX CoreAdmin API for standalone installations to be able to achieve the upgrade in a highly targeted and efficient way by only upgrading older segments and abstracting the setting of the merge policy. > Introduce a UPRADECOREINDEX CoreAdmin API to support index upgrade > -- > > Key: SOLR-18096 > URL: https://issues.apache.org/jira/browse/SOLR-18096 > Project: Solr > Issue Type: Improvement >Reporter: Rahul Goswami >Priority: Major > > Forking this JIRA from parent SOLR-17725 based on [~dsmiley]'s suggestion. > The merge policy (LatestVersionMergePolicy) part of the overall enhancement > made it into Solr 10 which enables uses to still be able to upgrade their > index for the next Solr version (Solr 11), although with more write overhead. > The other part of the enhancement is this JIRA which proposes exposing an > UPRADECOREINDEX CoreAdmin API for standalone installations to be able to > achieve the upgrade in a highly targeted and efficient way by only upgrading > older segments and abstracting the setting of the merge policy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
