This is an automated email from the ASF dual-hosted git repository.
dsmiley pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/solr.git
The following commit(s) were added to refs/heads/main by this push:
new b53ec648573 Documentation for SolrCloud update consistency (#4716)
b53ec648573 is described below
commit b53ec64857303583ec2565b65ff2440b630f6d93
Author: David Smiley <[email protected]>
AuthorDate: Mon Aug 24 23:56:58 2026 -0400
Documentation for SolrCloud update consistency (#4716)
Co-authored-by: Claude Fable 5 <[email protected]>
---
AGENTS.md | 20 ++
dev-docs/distributed-update-internals.adoc | 215 +++++++++++++++++++++
.../modules/deployment-guide/deployment-nav.adoc | 1 +
.../solrcloud-recoveries-and-write-tolerance.adoc | 1 +
.../pages/solrcloud-shards-indexing.adoc | 1 +
.../pages/solrcloud-update-consistency.adoc | 101 ++++++++++
6 files changed, 339 insertions(+)
diff --git a/AGENTS.md b/AGENTS.md
index ab22fdb9e5e..078406dfb74 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -52,6 +52,26 @@ While README.md and CONTRIBUTING.md are mainly written for
humans, this file is
- New classes should have some javadocs
- Changes should not have code comments communicating the change, which are
instead great comments to leave for code review / commentary
+## Developer Docs Index
+
+Before diving into code on these topics, read the matching doc in `dev-docs/`.
When adding a new dev doc, add a line here.
+
+Internals:
+
+- `dev-docs/overseer/overseer.adoc` — Overseer: cluster state updates,
ZkStateWriter, collection API message flow
+- `dev-docs/shard-split/shard-split.adoc` — SPLITSHARD: shard/replica states,
tlog buffering during split
+- `dev-docs/distributed-update-internals.adoc` — SolrCloud update path:
routing, `_version_`/optimistic concurrency, tlog durability, replication acks,
shard terms (user-facing consistency model: ref-guide page
`solrcloud-update-consistency.adoc`)
+- `dev-docs/plugins-modules-packages.adoc` — plugin/module/package concepts
+- `dev-docs/apis.adoc`, `dev-docs/v2-api-conventions.adoc` — API design and v2
conventions
+- `dev-docs/ui/` — new Admin UI architecture, component development, testing
+
+Process & tooling:
+
+- `dev-docs/solr-source-code.adoc`, `git.adoc`, `IDEs.adoc`, `jvms.adoc` —
build and dev environment
+- `dev-docs/ref-guide/` — ref-guide authoring (AsciiDoc syntax, Antora
templates)
+- `dev-docs/dependency-upgrades.adoc`, `lucene-upgrade.md`,
`working-between-major-versions.adoc` — upgrades and branch management
+- `dev-docs/releasing.adoc`, `changelog.adoc`, `asf-jenkins.adoc` — release
and CI process
+
## Changelog
- We use the "logchange" tooling to manage our changelog. See
`dev-docs/changelog.adoc` for details and conventions
diff --git a/dev-docs/distributed-update-internals.adoc
b/dev-docs/distributed-update-internals.adoc
new file mode 100644
index 00000000000..777feaa748c
--- /dev/null
+++ b/dev-docs/distributed-update-internals.adoc
@@ -0,0 +1,215 @@
+= Distributed Update Internals (SolrCloud)
+:toc: macro
+:toclevels: 3
+
+toc::[]
+
+== Why this doc
+
+The Solr Reference Guide states the user-facing consistency model of SolrCloud
updates in
+https://github.com/apache/solr/blob/main/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc[SolrCloud
Update Consistency Model].
+This document explains *how* those guarantees are implemented: the distributed
update path from the receiving node through the shard leader to the replicas,
and the versioning scheme that everything else leans on.
+
+Class names below are under `solr/core/src/java/org/apache/solr/` unless
otherwise noted.
+
+== Request lifecycle
+
+An `/update` request is parsed by a loader into a stream of `AddUpdateCommand`
/ `DeleteUpdateCommand` / `CommitUpdateCommand` objects, each fed through the
update request processor (URP) chain.
+Everything discussed here happens inside two processors near the end of that
chain: `update.processor.DistributedZkUpdateProcessor` (subclass of
`DistributedUpdateProcessor`; "DUP" below) and
`update.processor.RunUpdateProcessor`.
+URPs configured *before* the DUP run only on the node that received the client
request; URPs *after* it (and `RunUpdateProcessor`) run on the leader and again
on every replica.
+
+[source,mermaid]
+----
+sequenceDiagram
+ participant C as Client
+ participant N as Receiving node
+ participant L as Shard leader
+ participant R as NRT/TLOG replicas
+
+ C->>N: /update (batch of docs)
+ N->>L: forward each doc (DistribPhase=TOLEADER)
+ L->>L: per-doc lock, OCC check, assign _version_
+ L->>L: Lucene write + tlog append
+ L--)R: stream doc (DistribPhase=FROMLEADER, async)
+ R->>R: drop if stale version, else apply + tlog
+ L->>L: finish(): await replica responses,<br/>demote failed replicas via
shard terms,<br/>flush tlog
+ L-->>N: per-request response (+ achieved rf)
+ N-->>C: HTTP 200 (or error)
+----
+
+=== Routing: setupRequest
+
+`DistributedZkUpdateProcessor.setupRequest()` decides, per document, what role
this node plays.
+It computes the target slice via the collection's `DocRouter`, looks up the
shard leader with `ZkStateReader.getLeaderRetry(...)`, and compares it to the
local core.
+The `DISTRIB_UPDATE_PARAM` (`update.distrib`) carries a `DistribPhase` marking
where the request came from:
+
+* `NONE` — an external client request; if this node is not the leader, set
`forwardToLeader` and target a single `SolrCmdDistributor.ForwardNode` (the
leader).
+* `TOLEADER` — forwarded from a peer; this node is (should be) the leader;
compute the replica fan-out list.
+* `FROMLEADER` — forwarded from the leader; apply locally only, no further
distribution.
+
+The fan-out list comes from `getReplicaNodesForLeader(...)`: only `NRT` and
`TLOG` replicas (PULL replicas never receive updates), excluding down/non-live
replicas and any replica whose shard term is already behind the leader's
(`ZkShardTerms.skipSendingUpdatesTo`) — those are collected in
`skippedCoreNodeNames` and will have their terms pushed further down at request
end.
+
+Retry limits for internal hops are asymmetric: forwarding to a leader retries
up to `solr.retries.on.forward` (default 25) times, while a leader sending to
followers retries only `solr.retries.to.followers` (default 3) times — a
follower that can't take the update is demoted instead of retried hard.
+
+=== Leader-side processing
+
+For an add, `DistributedUpdateProcessor.versionAdd` runs the whole
per-document decision inside `UpdateLocks.runWithLock(id, ...)` — see
<<UpdateLocks>>.
+`leaderLogic` is true when this core is the leader and the command is not a
replay/peer-sync (`leaderLogicWithVersionIntegrityCheck`); a non-leader
receiving an update *without* a version is rejected as an invalid state.
+
+On the leader:
+
+1. If the command carries a client-supplied version constraint, perform the
optimistic-concurrency check (see <<Leader-side optimistic concurrency>>).
+2. If the command is an atomic (partial) update, resolve it into a full
replacement document (`getUpdatedDocument` → `AtomicUpdateDocumentMerger`),
reading the current document through `RealTimeGetComponent.getInputDocument` —
which sees uncommitted state via the tlog.
+3. Assign a new version: `cmd.setVersion(vinfo.getNewClock())` (see <<The
_version_ clock>>).
+4. Apply locally: `doLocalAdd` → `RunUpdateProcessor` →
`DirectUpdateHandler2.addDoc` (Lucene `updateDocument`) and `UpdateLog.add`
(tlog append).
+5. Hand the (now fully-resolved, versioned) document to `doDistribAdd` for
fan-out.
+
+Because steps 1–4 happen under the per-document lock, concurrent updates to
the same id are serialized at the leader, read-modify-write atomic updates are
linearizable per document, and version assignment order matches apply order per
document.
+
+=== Fan-out: SolrCmdDistributor and StreamingSolrClients
+
+`update.SolrCmdDistributor.distribAdd/distribDelete` submits each command to
the target nodes.
+Ordinary adds/deletes are *fire-and-forget at the per-document level*:
`update.StreamingSolrClients` maintains one `ConcurrentUpdateJettySolrClient`
per destination URL (queue size 100, deliberately low thread count — the class
comments that more threads "could cause updates to be reordered on a greater
scale"), and the document is queued into its stream without waiting for a
response.
+
+Exceptions that are sent synchronously (blocking per command): in-place
updates (a dependent in-place update must not overtake its predecessor in a
stream), and forwards to sub-shard leaders or routing-rule targets during shard
split / migrate.
+
+The leader therefore does not know a document's replica outcome at the time it
processes the next document.
+All outcomes are collected at request end.
+
+=== Request finish: acknowledgment and error triage
+
+When the loader has fed all commands, `DistributedUpdateProcessor.finish()`
runs `doDistribFinish()` and then `RunUpdateProcessor.finish()`.
+`DistributedZkUpdateProcessor.doDistribFinish()` is where the acknowledgment
semantics live:
+
+1. If this leader changed its index and skipped any known-stale replicas, bump
terms now: `ZkShardTerms.ensureTermsIsHigher(leader, skippedCoreNodeNames)`.
+2. `cmdDistrib.finish()` — *block* until every queued replica request has
completed (this is the only wait for replica responses in the whole path).
+3. Walk `cmdDistrib.getErrors()` and triage:
+ * Error on a `ForwardNode` (this node → leader): added to
`errorsForClient`; the client sees the failure and may retry.
+ * Error on a `StdNode` (leader → follower): *not* a client error — the code
comments "for now we don't error - we assume if it was added locally, we
succeeded".
+ Unless the error is a commit (`commit_end_point` requests never trigger
recovery) the follower's coreNodeName is collected for demotion — after
double-checking against ZK that we are still the leader and the errored node is
still one of our replicas.
+ * Special case: if the remote error's metadata says `cause=LeaderChanged`
(SOLR-6511 — the "follower" now believes it is the leader), the error *is*
propagated to the client so it can retry against the new leader.
+4. `ensureTermsIsHigher(leader, replicasShouldBeInLowerTerms)` — the demotion.
+ The demoted replica's term watcher notices it is behind and puts the core
into recovery.
+ This term mechanism (SOLR-11702) is the replacement for the old znode-based
"leader-initiated recovery" (LIR); no LIR znodes exist anymore.
+5. Compute the achieved replication factor: each shard leader counts itself
plus each follower that acked (`LeaderRequestReplicationTracker`), the
originating node takes the minimum across shards
(`RollupRequestReplicationTracker`), and the result is reported as `rf` in the
response header.
+ It is purely informational; a top-of-class TODO ("optionally fail if n
replicas are not reached...") records the unimplemented alternative.
+
+Client-visible errors are aggregated into `DistributedUpdatesAsyncException`
(status: the common code if all agree, else 400 if all 4xx, else 500).
+
+Finally `RunUpdateProcessor.finish()` calls `UpdateLog.finish(null)` — the
per-request tlog flush described next.
+
+=== Durability: UpdateLog and TransactionLog
+
+`update.UpdateLog` owns a current `update.TransactionLog` (tlog) plus recent
old ones, and an in-memory map from doc id to a `LogPtr` into the tlog — the
map that makes uncommitted documents visible to realtime get and to
atomic-update resolution.
+
+Writes: `UpdateLog.add/delete/deleteByQuery` append a record to the current
tlog through a buffered `FastOutputStream` over a file channel.
+*Nothing is flushed per document.*
+Durability happens per *request*, in `TransactionLog.finish(syncLevel)`:
+
+* `NONE` — do nothing.
+* `FLUSH` (the default) — flush the JVM buffer to the OS; survives a JVM crash
/ `kill -9`, but not an OS crash or power loss.
+* `FSYNC` — additionally `channel.force(true)`.
+ The fsync is deliberately outside the buffer lock; the code notes a partial
last record after power failure is expected and tolerated by the reader.
+
+`syncLevel` is configured on `<updateLog>` in `solrconfig.xml`.
+So the ref-guide statement "documents are written to the tlog before the
indexing call returns" is true, but with default `FLUSH` the response does not
imply the bytes reached the disk platter.
+
+Commits rotate the tlog: `UpdateLog.preCommit` starts a new tlog (so the old
one is definitely fully covered by the index commit), and `postCommit` writes a
commit marker into the old one.
+On startup, `UpdateLog.recoverFromLog()` replays any tlog tail not covered by
a commit — this is what makes acked-but-uncommitted updates survive a restart.
+Retention is bounded by `numRecordsToKeep` (default 100) and
`maxNumLogsToKeep` (default 10), which also bound how far a replica can fall
behind before PeerSync is impossible and full replication is required.
+
+The UpdateLog also has a state machine (`ACTIVE`, `BUFFERING`,
`APPLYING_BUFFERED`, `REPLAYING`) used during recovery and shard split: while a
core is recovering, incoming `FROMLEADER` updates are written to a separate
buffer tlog *without* being applied, and replayed at the end
(`applyBufferedUpdates`).
+State transitions quiesce all in-flight updates through
`UpdateLocks.blockUpdates()` (the write side of a fair read/write lock; every
normal update holds the read side).
+See `dev-docs/shard-split/shard-split.adoc` for the shard-split use of
buffering.
+
+=== Replica-side processing
+
+A replica receiving `DistribPhase.FROMLEADER` runs the same
`versionAdd`/`versionDelete` but with `leaderLogic == false`:
+
+* An update without a `\_version_` is rejected (`missing _version_ on update
from leader`) — replicas never mint versions.
+* If the local UpdateLog is not `ACTIVE` (the core is recovering), the update
is written to the buffer tlog and dropped (no index write).
+* Otherwise the *drop rule* runs — the single check that makes asynchronous,
possibly-reordered delivery safe:
++
+[source,java]
+----
+Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId());
+if (lastVersion != null && Math.abs(lastVersion) >= versionOnUpdate) {
+ // This update is a repeat, or was reordered. We need to drop this update.
+ return true;
+}
+----
++
+Application on a replica is therefore idempotent (repeats are dropped) and
order-insensitive *per document* (an older version arriving late is dropped).
+Nothing orders updates across different documents.
+* On a TLOG replica (not currently leader), the command additionally gets
`UpdateCommand.IGNORE_INDEXWRITER`: it is recorded in the tlog but not indexed
— the index arrives later by segment replication, and the tlog exists so the
replica can replay it if elected leader.
+
+Deletes store *negative* versions (hence the `Math.abs`), letting a version
lookup distinguish "deleted at version v" from "exists at version v" while
still ordering both.
+
+Two reorder edge cases get dedicated machinery:
+
+* *Delete-by-query*: on the leader, `versionDeleteByQuery` runs under
`UpdateLocks.blockUpdates()` — DBQ quiesces *all* updates on the core, because
it can affect any document.
+ Replicas keep a list of recent DBQs and re-execute them over an add that
arrives out of order relative to the DBQ.
+ A DBQ is also fanned out from the originating node to *all* shard leaders,
and is not atomic across shards.
+* *In-place updates* carry `distrib.inplace.prevversion`; a replica that has
not yet seen that previous version waits for it (`waitForDependentUpdates`,
using the per-doc lock's `Condition`), and if it never arrives fetches the full
document from the leader (`fetchFullUpdateFromLeader`).
+
+== Versioning and optimistic concurrency
+
+=== The _version_ clock
+
+`update.VersionInfo.getNewClock()` implements a time-based Lamport clock,
synchronized per core:
+
+[source,java]
+----
+long time = System.currentTimeMillis();
+long result = time << 20;
+if (result <= vclock) {
+ result = vclock + 1;
+}
+vclock = result;
+----
+
+Properties that matter:
+
+* Strictly increasing per core, so per-document last-writer-wins is well
defined under a single leader.
+* Wall-clock based so that a restarted or newly elected leader (with an empty
in-memory clock) does not go back in time relative to versions already in the
index — correctness across leader changes leans on cluster clocks being roughly
synchronized.
+ The low 20 bits are a same-millisecond counter (~1M versions/ms before the
clock runs ahead of real time).
+* Not contiguous — a commented-out pure-counter alternative in `VersionInfo`
notes contiguous versions would make missing-update detection easier; Solr
instead detects gaps via PeerSync's version-list exchange.
+
+The `\_version_` field must exist in the schema, single-valued,
indexed-or-docValues and stored-or-docValues
(`VersionInfo.getAndCheckVersionField`).
+It must be assigned by Solr internally: user-supplied values would break the
replica drop rule.
+(Use `DocBasedVersionConstraintsProcessorFactory` for application-level
version fields.)
+
+=== Where the version constraint comes from
+
+`versionOnUpdate` is taken, in priority order, from the command itself, the
document's `\_version_` field, or the `\_version_` request parameter.
+On internal `FROMLEADER` hops it is the leader-assigned version; on client
requests it is the client's optimistic-concurrency constraint (0 when absent).
+One subtlety: a leader receiving a document forwarded from *another
collection* (`distrib.from.collection`, the MIGRATE path) discards the incoming
version and stamps its own.
+
+=== Leader-side optimistic concurrency
+
+When a client supplies a nonzero version, the leader checks it against
`vinfo.lookupVersion(id)` (tlog first, then index — uncommitted state counts)
before assigning the new version:
+
+* `> 1` — must equal the current version exactly.
+* `1` — the document must exist (any positive current version).
+* `< 0` — the document must not exist.
+* `0` — no check.
+
+A failed check raises `ErrorCode.CONFLICT` (HTTP 409) — unless
`failOnVersionConflicts=false`, which silently drops the update instead (useful
for batch loads where any conflicting doc should just be skipped).
+The check runs only on the leader, under the per-document lock, so it is
atomic with the version assignment: two clients doing conditional updates on
the same document cannot both win.
+
+=== UpdateLocks
+
+`update.UpdateLocks` (SOLR-14679) replaced the historical fixed-size
`VersionBucket` striping.
+It keeps a hash-keyed map of pooled, fair `ReentrantLock`+`Condition` pairs
with refcounting, so a lock exists only while some thread is operating on that
document id, plus the global `blockUpdatesLock` read/write lock described
earlier.
+Lock acquisition times out after `docLockTimeoutMs`, surfacing pathological
contention as an error rather than a hang.
+
+Everything that must be atomic per document happens inside `runWithLock`: the
OCC check, atomic-update read-modify-write, version assignment, tlog append and
index write.
+This is also what makes realtime get reliable — a concurrent RTG cannot
observe a state between the tlog map update and the version assignment.
+
+== References
+
+* Ref guide:
https://github.com/apache/solr/blob/main/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc[SolrCloud
Update Consistency Model] (the user-facing contract), plus the pages on shards
and indexing, recoveries and write tolerance, commits and transaction logs,
partial document updates, and realtime get.
+* `dev-docs/shard-split/shard-split.adoc` — tlog buffering during shard split.
+* JIRA: https://issues.apache.org/jira/browse/SOLR-11702[SOLR-11702] (shard
terms replace LIR),
https://issues.apache.org/jira/browse/SOLR-14679[SOLR-14679] (`UpdateLocks`
replaces version buckets),
https://issues.apache.org/jira/browse/SOLR-6511[SOLR-6511] (`LeaderChanged`
propagation), https://issues.apache.org/jira/browse/SOLR-7141[SOLR-7141]
(recovery vs. in-flight updates).
+
+Not covered here (candidates for future documents): replica-type internals
(NRT/TLOG/PULL), recovery and leader election (`RecoveryStrategy`, `PeerSync`,
`ZkShardTerms` invariants, `leaderVoteWait`), and commit/visibility internals.
diff --git a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc
b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc
index 04dfeb76643..83bb2e286ed 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc
@@ -33,6 +33,7 @@
** SolrCloud Clusters
*** xref:solrcloud-shards-indexing.adoc[]
*** xref:solrcloud-recoveries-and-write-tolerance.adoc[]
+*** xref:solrcloud-update-consistency.adoc[]
*** xref:solrcloud-distributed-requests.adoc[]
*** xref:node-roles.adoc[]
*** xref:aliases.adoc[]
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc
b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc
index 5980a4ebfaf..fc387be258e 100644
---
a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc
+++
b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc
@@ -17,6 +17,7 @@
// under the License.
SolrCloud is highly available and fault tolerant in reads and writes.
+For a precise statement of what a successful update does and does not
guarantee, see xref:solrcloud-update-consistency.adoc[].
== Write Side Fault Tolerance
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc
b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc
index 622875b99f8..97a960f9895 100644
---
a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc
+++
b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc
@@ -27,6 +27,7 @@ A different collection might simply use a "hash" on the
uniqueKey of each docume
There is support for distributing both the index process and the queries
automatically, and ZooKeeper provides failover and load balancing.
As well as supporting replication, automatic index splitting into shards,
there is support for automatic routing of documents to specific shards by a
sharding strategy.
Additionally, every shard can have multiple replicas for additional robustness.
+The guarantees that distributed indexing provides are stated in
xref:solrcloud-update-consistency.adoc[].
== Leaders and Replicas
diff --git
a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc
b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc
new file mode 100644
index 00000000000..345d0b15389
--- /dev/null
+++
b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc
@@ -0,0 +1,101 @@
+= SolrCloud Update Consistency Model
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+This page states the consistency model of SolrCloud update requests
(`/update`): what a successful response guarantees, and what it deliberately
does not.
+Several of these guarantees are intentionally weaker than those of a
transactional database; the trade-offs favor indexing throughput and
availability.
+
+Developers interested in how these guarantees are implemented should read the
companion document
https://github.com/apache/solr/blob/main/dev-docs/distributed-update-internals.adoc[Distributed
Update Internals] in the source repository.
+
+== Replication
+
+An update request may be sent to any node in the cluster.
+Each document is routed to the leader replica of its shard, which assigns the
document a version, applies the update locally, and forwards it to the other
in-sync NRT and TLOG replicas of the shard.
+PULL replicas never receive individual updates; they periodically copy index
segments from the leader.
+
+A successful (HTTP 200) response guarantees that every document in the request
was applied and written to the
xref:configuration-guide:commits-transaction-logs.adoc#transaction-log[transaction
log] on its shard leader, and that Solr attempted to replicate it to the
shard's other in-sync replicas.
+
+It does *not* guarantee that any replica other than the leader applied the
update.
+By design, a replica that fails to acknowledge an update does not fail the
request; instead the leader marks that replica as out-of-date, and the replica
must recover (catch up from the leader) before serving queries or becoming a
leader.
+Clients that need to know how widely an update was replicated can inspect the
achieved replication factor (`rf`) in the response header; it is informational
only and is never enforced.
+See xref:solrcloud-recoveries-and-write-tolerance.adoc[] for details.
+
+== Durability
+
+Every update is written to the leader's transaction log before the response is
returned, and replayed on startup if the node was stopped before a hard commit.
+With the default transaction log `syncLevel` of `flush`, this survives a JVM
crash or process kill but not necessarily an operating system crash or power
loss; configure `syncLevel` to `fsync` to close that gap at some cost to
throughput.
+Index files themselves are durable once a hard commit completes.
+See xref:configuration-guide:commits-transaction-logs.adoc[] for configuration
details.
+
+An acknowledged update that has reached only the leader can still be lost if
the leader is permanently destroyed before any replica receives it.
+The durability floor of an acknowledged update is therefore one node, unless
the client verifies `rf`.
+
+== Ordering
+
+Updates to the *same document* are strictly ordered.
+The shard leader serializes concurrent updates to a given document id and
assigns each a monotonically increasing `\_version_`; replicas ignore any
update older than the version they already have.
+The last write accepted by the leader wins.
+
+There is no ordering guarantee *across different documents*.
+Documents sent in one batch may be applied on replicas, and become searchable,
in a different order than submitted — especially across shards.
+
+== Atomicity
+
+The unit of atomicity is a single document.
+A document update fully replaces the previous version of that document
(xref:indexing-guide:partial-document-updates.adoc#atomic-updates[atomic
updates] are read-modify-write operations performed on the leader, producing a
full replacement document).
+Queries never observe a partially updated document.
+A xref:indexing-guide:indexing-nested-documents.adoc[nested document]
hierarchy counts as one document for this purpose: the root and all of its
descendants are indexed together as a single block, and an update to any
document in the hierarchy rewrites the entire block.
+xref:indexing-guide:partial-document-updates.adoc#in-place-updates[In-place
updates] are a narrow exception to full replacement — they modify eligible
docValues fields without reindexing the document — but they are still atomic
per document.
+
+There are no multi-document transactions.
+A batch of documents is processed as independent operations: some may succeed
while others fail, and there is no rollback of the documents that succeeded.
+By default a document error fails the request at that point; the
{solr-javadocs}/core/org/apache/solr/update/processor/TolerantUpdateProcessorFactory.html[`TolerantUpdateProcessorFactory`]
can be configured to instead continue past per-document failures and report
them all in the response.
+Similarly, a delete-by-query spanning multiple shards is not atomic across
those shards.
+The
xref:indexing-guide:indexing-with-update-handlers.adoc#rollback-operations[rollback
command] is not supported in SolrCloud.
+
+== Visibility
+
+An update is not searchable until a commit opens a new searcher; durability
(via the transaction log and hard commits) and searchability are independent.
+Each replica opens its searcher independently, so a document may briefly be
searchable on one replica but not another; applications using `autoSoftCommit`
or `commitWithin` must embrace this eventual consistency, as described in
xref:solrcloud-shards-indexing.adoc#ignoring-commits-from-client-applications-in-solrcloud[Ignoring
Commits from Client Applications in SolrCloud].
+An application can avoid replica-to-replica visibility differences from any
single user by routing that user's queries to the same replica, using stable
routing with the
xref:solrcloud-distributed-requests.adoc#shards-preference-parameter[`shards.preference`
parameter] (e.g., `replica.base:stable:hash:sessionId`); each user then
observes one replica's consistent timeline.
+
+The exception is xref:configuration-guide:realtime-get.adoc[RealTime Get],
which retrieves the latest version of a document by id — including uncommitted
updates — directly from the transaction log.
+
+== Optimistic Concurrency
+
+Solr supports conditional updates through the `\_version_` field: a client may
require that a document already exist, not exist, or exist at an exact version,
and a failed condition rejects the update with an HTTP 409 conflict.
+This is the supported way to prevent lost updates when multiple clients read,
modify, and rewrite the same documents.
+See
xref:indexing-guide:partial-document-updates.adoc#optimistic-concurrency[Optimistic
Concurrency].
+
+Applications whose versions originate in an external system of record can
instead enforce ordering with a version field of their own, via
`DocBasedVersionConstraintsProcessorFactory`; see
xref:indexing-guide:partial-document-updates.adoc#document-centric-versioning-constraints[Document
Centric Versioning Constraints].
+
+== Failures and Retries
+
+Failures that are visible to the client include: the receiving node being
unable to reach a shard leader, a shard having no elected leader, a leader
change detected mid-request, version conflicts, and per-document errors such as
schema violations.
+Failed requests may leave earlier documents of the same batch applied.
+
+Retrying a failed request is safe for full-document adds and deletes-by-id:
reapplying the same document produces the same end state (though a retry can
overwrite a newer concurrent write from another client, as with any
last-write-wins system).
+Retrying is *not* inherently safe for atomic updates that are not idempotent,
such as `inc`ing a counter or `add`ing a list value, because the original
attempt may have been applied even though the response was lost.
+Use optimistic concurrency to make such read-modify-write cycles safe to retry.
+
+== Leader Failover
+
+When a leader fails, Solr elects a new leader from the replicas that are known
to be up-to-date, and the new leader first syncs any recent updates from its
peers.
+Acknowledged updates therefore survive leader failover whenever at least one
up-to-date replica remains.
+
+If no up-to-date replica is available after a waiting period
(`leaderVoteWait`), Solr chooses availability over consistency: a potentially
stale replica becomes leader so the shard can continue accepting updates, and
updates that only the old leader had may be lost.
+Solr logs a "potential data loss" warning when this happens.
