[
https://issues.apache.org/jira/browse/SOLR-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Goswami updated SOLR-18190:
---------------------------------
Description:
*+Objective+*
Expose index-upgrade functionality at collection scope in SolrCloud as a new
"UPGRADECOLLECTIONINDEX" Collections API command with async support and
`REQUESTSTATUS` tracking.
*+Approach+*
_Write freeze_ *+* _hybrid local upgrade_ - Collection is set to `readOnly` for
the duration. Each replica type is upgraded via its designed index-update
mechanism. Which means
* Leader is upgraded using the CoreAdmin UPGRADECOREINDEX API that is now
available.
* NRT replicas are also individually upgraded using the same UPGRADECOREINDEX
API
* TLOG/PULL replicas
Why not upgrade only the leader and rely on distributed forwarding to NRT
replicas ? `DistributedZkUpdateProcessor` enforces the collection-level
`readOnly` on every node, including replicas receiving forwarded updates.
Forwarding is blocked by the same write freeze that protects against external
writes.
+*Operational Flow*+
1. Coordinator sets `readOnly=true` on the collection via `MODIFYCOLLECTION`
(blocks all external writes at `DistributedZkUpdateProcessor`).
2. For each shard (sequentially):
a. Identify the current leader via `getLeaderRetry()`.
b. {*}{{*}}Upgrade the leader{{*}}{*} via CoreAdmin `UPGRADECOREINDEX` with
`cloudMode=true`. The leader uses a stripped-down chain (`LogUpdateProcessor` →
`RunUpdateProcessor`, no `DistributedUpdateProcessor`) to rewrite old segments
locally. No version reassignment, no distributed forwarding. The original
`{_}version{_}` is preserved both in the indexed document and on the
`AddUpdateCommand` (for tlog consistency). After rewriting, the original merge
policy is restored before commit, followed by `expungeDeletes` to clean
tombstone segments.
c. {*}{{*}}Upgrade NRT non-leader replicas{{*}}{*} in parallel using the same
mechanism. Each NRT replica independently rewrites its own segments with zero
network I/O.
d. {*}{{*}}TLOG/PULL replicas{{*}}{*} converge via their normal background
replication from the now-upgraded leader.
e. {*}{{*}}Convergence polling{{*}}{*}: Poll all replicas with `checkOnly=true`
until every replica reports no old-format segments remaining (see Convergence
Polling below).
3. Clear `readOnly=false` only after all shards validate. On any failure, the
collection remains read-only for operator intervention.
*+Limitations+*
- Nested documents: Not supported (existing limitation in `UpgradeCoreIndex`).
- Write downtime: The collection is unavailable for writes for the duration of
the upgrade. Zero write downtime may be revisited after solving the DBQ
resurrection problem.
- Leader election resilience: If a leader election occurs during the upgrade,
progress may be lost and the command must be re-run. This should be fine since
the operation is designed to be idempotent.
- Co-located replica IO: NRT replicas on the same node are upgraded in
parallel, which may cause IO contention. Node-aware throttling deferred to a
future version.
was:
*+Objective+*
Expose index-upgrade functionality at collection scope in SolrCloud as a new
"UPGRADECOLLECTIONINDEX" Collections API command with async support and
`REQUESTSTATUS` tracking.
*+Approach+*
_Write freeze_ *+* _hybrid local upgrade_ - Collection is set to `readOnly` for
the duration. Each replica type is upgraded via its designed index-update
mechanism. Which means
* Leader is upgraded using the CoreAdmin UPGRADECOREINDEX API that is now
available.
* NRT replicas are also individually upgraded using the same UPGRADECOREINDEX
API
* TLOG/PULL replicas
Why not upgrade only the leader and rely on distributed forwarding to NRT
replicas ? `DistributedZkUpdateProcessor` enforces the collection-level
`readOnly` on every node, including replicas receiving forwarded updates.
Forwarding is blocked by the same write freeze that protects against external
writes.
+*Operational Flow*+
1. Coordinator sets `readOnly=true` on the collection via `MODIFYCOLLECTION`
(blocks all external writes at `DistributedZkUpdateProcessor`).
2. For each shard (sequentially):
a. Identify the current leader via `getLeaderRetry()`.
b. *{*}Upgrade the leader{*}* via CoreAdmin `UPGRADECOREINDEX` with
`cloudMode=true`. The leader uses a stripped-down chain (`LogUpdateProcessor` →
`RunUpdateProcessor`, no `DistributedUpdateProcessor`) to rewrite old segments
locally. No version reassignment, no distributed forwarding. The original
`{_}version{_}` is preserved both in the indexed document and on the
`AddUpdateCommand` (for tlog consistency). After rewriting, the original merge
policy is restored before commit, followed by `expungeDeletes` to clean
tombstone segments.
c. *{*}Upgrade NRT non-leader replicas{*}* in parallel using the same
mechanism. Each NRT replica independently rewrites its own segments with zero
network I/O.
d. *{*}TLOG/PULL replicas{*}* converge via their normal background replication
from the now-upgraded leader.
e. *{*}Convergence polling{*}*: Poll all replicas with `checkOnly=true` until
every replica reports no old-format segments remaining (see Convergence Polling
below).
3. Clear `readOnly=false` only after all shards validate. On any failure, the
collection remains read-only for operator intervention.
*+Limitations+*
- Nested documents: Not supported (existing limitation in `UpgradeCoreIndex`).
- Write downtime: The collection is unavailable for writes for the duration of
the upgrade. Zero write downtime may be revisited after solving the DBQ
resurrection problem.
- Leader election resilience: If a leader election occurs during the upgrade,
progress may be lost and the command must be re-run.
- Co-located replica IO: NRT replicas on the same node are upgraded in
parallel, which may cause IO contention. Node-aware throttling deferred to a
future version.
> Collection-Level Index Upgrade API in SolrCloud (UPRGADECOLLECTIONINDEX)
> ------------------------------------------------------------------------
>
> Key: SOLR-18190
> URL: https://issues.apache.org/jira/browse/SOLR-18190
> Project: Solr
> Issue Type: Improvement
> Reporter: Rahul Goswami
> Priority: Major
>
> *+Objective+*
> Expose index-upgrade functionality at collection scope in SolrCloud as a new
> "UPGRADECOLLECTIONINDEX" Collections API command with async support and
> `REQUESTSTATUS` tracking.
> *+Approach+*
> _Write freeze_ *+* _hybrid local upgrade_ - Collection is set to `readOnly`
> for the duration. Each replica type is upgraded via its designed index-update
> mechanism. Which means
> * Leader is upgraded using the CoreAdmin UPGRADECOREINDEX API that is now
> available.
> * NRT replicas are also individually upgraded using the same
> UPGRADECOREINDEX API
> * TLOG/PULL replicas
> Why not upgrade only the leader and rely on distributed forwarding to NRT
> replicas ? `DistributedZkUpdateProcessor` enforces the collection-level
> `readOnly` on every node, including replicas receiving forwarded updates.
> Forwarding is blocked by the same write freeze that protects against external
> writes.
> +*Operational Flow*+
> 1. Coordinator sets `readOnly=true` on the collection via `MODIFYCOLLECTION`
> (blocks all external writes at `DistributedZkUpdateProcessor`).
> 2. For each shard (sequentially):
> a. Identify the current leader via `getLeaderRetry()`.
> b. {*}{{*}}Upgrade the leader{{*}}{*} via CoreAdmin `UPGRADECOREINDEX` with
> `cloudMode=true`. The leader uses a stripped-down chain (`LogUpdateProcessor`
> → `RunUpdateProcessor`, no `DistributedUpdateProcessor`) to rewrite old
> segments locally. No version reassignment, no distributed forwarding. The
> original `{_}version{_}` is preserved both in the indexed document and on the
> `AddUpdateCommand` (for tlog consistency). After rewriting, the original
> merge policy is restored before commit, followed by `expungeDeletes` to clean
> tombstone segments.
> c. {*}{{*}}Upgrade NRT non-leader replicas{{*}}{*} in parallel using the same
> mechanism. Each NRT replica independently rewrites its own segments with zero
> network I/O.
> d. {*}{{*}}TLOG/PULL replicas{{*}}{*} converge via their normal background
> replication from the now-upgraded leader.
> e. {*}{{*}}Convergence polling{{*}}{*}: Poll all replicas with
> `checkOnly=true` until every replica reports no old-format segments remaining
> (see Convergence Polling below).
> 3. Clear `readOnly=false` only after all shards validate. On any failure, the
> collection remains read-only for operator intervention.
> *+Limitations+*
> - Nested documents: Not supported (existing limitation in
> `UpgradeCoreIndex`).
> - Write downtime: The collection is unavailable for writes for the duration
> of the upgrade. Zero write downtime may be revisited after solving the DBQ
> resurrection problem.
> - Leader election resilience: If a leader election occurs during the
> upgrade, progress may be lost and the command must be re-run. This should be
> fine since the operation is designed to be idempotent.
> - Co-located replica IO: NRT replicas on the same node are upgraded in
> parallel, which may cause IO contention. Node-aware throttling deferred to a
> future version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
