Siyao Meng created HDDS-16925:
---------------------------------
Summary: SCM forgets all container replicas when it installs a
Ratis snapshot, until the next full container reports
Key: HDDS-16925
URL: https://issues.apache.org/jira/browse/HDDS-16925
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-8-keep-container-replicas-across-reinitialize.patch,
TestBugCR8ReplicaWipeOnReinitialize.java
h3. Mechanism
{{ContainerStateManagerImpl.reinitialize}}, which runs when an SCM installs a
checkpoint from the leader ({{SCMHAManagerImpl.startServices}}), replaces its
{{ContainerStateMap}} with an empty one and reloads only the {{ContainerInfo}}
rows from the installed DB. Replicas are not in the DB; they come from datanode
reports. So after the install the SCM knows no replica of any container,
although the datanodes still hold them all. They come back with incremental
reports for changed containers and with each datanode's next full container
report ({{hdds.container.report.interval}}, 60 minutes by default). Nothing
asks for an earlier full report, and safe mode is not entered again on an
install or on a leader change. Replica updates take only their stripe lock, not
the global lock that {{reinitialize}} holds, so updates that run during the
swap are lost as well.
Recon's SCM DB snapshot sync
({{ReconStorageContainerManagerFacade.initializeNewRdbStore}}) calls the same
method and wipes Recon's own replica view the same way; the sync is a Recon
admin operation and Recon sends no commands (by code reading, not run).
h3. Trigger
# A follower SCM that is already processing datanode reports installs a
checkpoint from the leader. With the default
{{ozone.scm.ha.ratis.log.purge.enabled}}=false the leader purges its log only
up to the lowest commit index of its peers, so a lagging follower normally
catches up from the log. It installs a checkpoint when log purge is enabled and
it falls behind the purged index, for example after a long pause. A newly
bootstrapped or restarted SCM is not affected, because it starts its datanode
server only after it has caught up.
# Within the next report interval, that SCM becomes leader (failover or
leadership transfer).
h3. Impact
While the new leader has not received the full reports (argued from the code
unless stated otherwise):
* {{getContainerWithPipeline}} for a closed container returns an empty
pipeline, because the pipeline of a closed container is built from its
replicas. Reads that need a fresh lookup fail.
* After {{hdds.scm.wait.time.after.safemode.exit}} (5 minutes),
ReplicationManager sees the containers as MISSING (no replica known) or
UNDER_REPLICATED (some added back by incremental reports), replicates or
reconstructs them and later deletes the extra copies as over replicated. A
DELETING container with no known replica is marked DELETED while datanodes
still hold it.
* Decommission and maintenance: the monitor takes the node's containers from
NodeManager (not reset) and their replicas from ContainerManager (reset). This
monitor is not delayed by {{hdds.scm.wait.time.after.safemode.exit}}: the new
leader resumes tracking the nodes in {{NodeDecommissionManager.onBecomeLeader}}
and checks them every {{ozone.scm.datanode.admin.monitor.interval}} (30
seconds), so it acts on the wiped view right after the failover (by code
reading). A Ratis container then holds the decommission until the reports
return. An EC container with no known replica is classified MISSING, not
UNDER_REPLICATED, so it counts as sufficiently replicated, and the node can
finish decommission or enter maintenance without its EC replicas being copied.
Shown in the reproduction with the real check handler; the monitor and the
failover were not run.
* Everything returns to normal after the full reports. I found no path to data
loss on its own.
h3. Reproduction
PASS on the unmodified source. [^TestBugCR8ReplicaWipeOnReinitialize.java]
({{hadoop-hdds/server-scm}}) builds a real {{ContainerStateManagerImpl}} on a
real SCM DB with a CLOSED Ratis THREE container and a CLOSED EC 3+2 container,
each with a replica on a DECOMMISSIONING datanode, and calls {{reinitialize}}
with the same container table. It then applies the per container decision of
{{DatanodeAdminMonitorImpl.checkContainersReplicatedOnNode}} with the real
{{RatisContainerReplicaCount}}, {{ECContainerReplicaCount}} and
{{ECReplicationCheckHandler}}. Before: the EC container is UNDER_REPLICATED and
holds the decommission. After: {{Ratis replicas 0 (was 3), EC replicas 0 (was
5)}}, the EC container is MISSING and passes the decommission check. A passing
test means the defect is present.
h3. Not covered by the patch
Containers created on the leader while the follower lagged still have no
replicas on the new SCM until the datanodes report them. A leader readiness
gate after an install (wait for full reports, or ask the datanodes for one
before acting as leader) would cover those, as a follow up.
h3. Patch
[^CR-8-keep-container-replicas-across-reinitialize.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad.
The patch keeps the replica sets of the containers that are still in the
installed table. Inside the existing global write lock it also takes the write
lock of every stripe, so a replica update cannot run during the swap and be
overwritten by the copied set. Containers that are no longer in the table lose
their replicas with them. Lock order is unchanged (global, then stripe). Report
processing on that SCM waits while the container table reloads, which already
blocks container reads. With the patch the reproduction fails because the
replicas are still there.
Covered by {{testReinitializeKeepsReplicasOfRemainingContainers}} and
{{testReplicaUpdateDuringReinitializeIsNotLost}} in the existing
{{TestContainerStateManager}}. Without the change both fail (empty replica
sets). Copying the replicas without the stripe locks fails the second test,
because a replica removed during the swap comes back. With the patch the
container, report handler and replica suites in {{hadoop-hdds/server-scm}} (140
tests), the Recon container suites (22 tests) and
{{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} (5 tests) pass,
and checkstyle is clean.
Found by code review of the SCM Ratis snapshot and checkpoint install paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. HDDS-16896 (safe mode
rules of a bootstrapped SCM) is related but a different mechanism. The attached
patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
