Siyao Meng created HDDS-16891:
---------------------------------
Summary: After an SCM restart or leader change, SCM has no pending
container allocations and grants again the slots reserved for containers not
yet written
Key: HDDS-16891
URL: https://issues.apache.org/jira/browse/HDDS-16891
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: TestBugMC8PendingAllocationFailover.java
h3. Mechanism
{{PendingContainerTracker}} keeps the reservations in memory, in the
{{TwoWindowBucket}} of each {{DatanodeInfo}}. Only the leader fills it:
{{ContainerManagerImpl.allocateContainer}} calls
{{PipelineManagerImpl.checkSpaceAndRecordAllocation}} before the replicated
{{containerStateManager.addContainer}}. Followers apply {{addContainer}}, so
they know the container, but they do not record its slot. On restart,
{{ContainerStateManagerImpl.initialize}} loads the containers from the SCM DB
and nothing records their slots either. Nothing records the slots when an SCM
becomes leader. Datanodes create a container on its first write, so until then
no storage report counts it as committed. After a leader change or a restart,
SCM therefore sees the slots of granted containers that are not written yet as
free.
Every SCM receives the replicated {{addContainer}} and the heartbeats and
container reports that datanodes send to all SCMs (followers run the report
handlers, which also remove pending entries). The gap is that only the leader
records the reservation, before submitting.
h3. Trigger
# The leader allocates container C1 on a pipeline whose datanodes have one slot
left. C1 is not written yet.
# Leadership moves to another SCM (transfer, crash, long pause), or the SCM
restarts. This includes a single SCM.
# Once out of safe mode, the leader allocates C2 on the same datanodes.
h3. Impact
A leader change or restart drops every unconfirmed reservation on every
datanode at once, as if all of them had aged out. Only reservations younger
than two roll intervals ({{ozone.scm.pending.container.roll.interval}}, default
5m) and not yet confirmed by a container report are affected, and only
datanodes that are down to their last slots can be over granted. The expected
effect is that the datanode refuses one of the containers for lack of space
when it is created. On a Ratis pipeline the failed create marks the pipeline's
{{ContainerStateMachine}} unhealthy, so the pipeline is closed and the in
flight writes on it are retried elsewhere. This is argued from the code and was
not run. Replica placement after the leader change reads the same empty
buckets; that variant was not run either.
h3. Reproduction
[^TestBugMC8PendingAllocationFailover.java] ({{hadoop-ozone/integration-test}})
starts a mini cluster with 3 SCMs, 1 OM and 3 datanodes. Each datanode volume
uses the existing {{MockSpaceUsageCheckFactory$HalfTera}} (512 GiB) and the
container size is 300 GiB, so each datanode has one slot. Only the admin calls
{{allocateContainer}} and {{transferLeadership}} are used. The leader grants C1
and holds one pending entry per datanode, and it refuses a second container.
After the transfer, once the new leader is ready and out of safe mode, it has
C1 OPEN with no replicas, committed space 0 and no pending entry on all three
datanodes, and it grants C2 on the same three datanodes. A passing test means
the defect is present (under a minute on an unmodified checkout). The restart
case was not run.
h3. Suggested fix
No patch is attached yet, so that the design can be agreed first. The proposal
is a rebuild of the buckets when an SCM becomes the serving leader:
* Run once the SCM is leader ready and out of safe mode, the point where
{{ReplicationManager}} starts. After a restart the datanodes are registered by
then, and allocation is refused in safe mode anyway.
* First drop all pending entries, then seed: for each OPEN container, record a
slot on each datanode of its pipeline that has no replica of it.
* Finish the rebuild before the first allocation is served.
This covers a leader change, a restart of one SCM or of all of them, a follower
that installed a Ratis snapshot, and a newly bootstrapped SCM. The drop also
clears stale entries of a returning leader: replication reservations that are
not removed when {{ReplicationManager}} clears its pending ops on leader
change, and the case below. It keeps the apply path free of node manager calls.
Its cost is that it also reserves for OPEN containers that the previous leader
had already aged out (for example containers {{getMatchingContainer}} opened
ahead of use), so for up to two roll intervals after the rebuild SCM grants
fewer containers than before, never more. A datanode that registers after the
rebuild starts with an empty bucket, as today.
The alternative, recording the slot whenever an SCM applies the replicated
{{addContainer}}, needs no scan, but it does not cover an SCM restart or a
snapshot install, because containers loaded from the SCM DB are not applied
again. It also puts a node manager call in the apply path (which Recon runs
too), has to avoid counting the leader's own entry twice, and on a lagging
follower the container report for a container can arrive before the follower
applies its {{addContainer}}, leaving the entry to age out. Waiting for a
container report interval after leader ready does not help either, because a
container that is not written yet is in no report.
A related case: when the replicated {{addContainer}} fails after the slot was
reserved, the reservation is kept. That is deliberate.
{{SCMRatisServerImpl.submitRequest}} gives up after
{{ozone.scm.ha.ratis.request.timeout}} while the entry stays in the Raft log,
and on step down Ratis (3.3.1) answers the requests already in the leader's log
with {{NotLeaderException}} although the next leader may commit them. A timeout
never says whether the entry will commit, and a step down reply has the same
result code ({{SCM_NOT_LEADER}}) as a definite rejection, so releasing on
failure could turn a bounded under grant into an over grant. Such an entry ages
out within two roll intervals, or is dropped by the rebuild above when the SCM
becomes leader again.
Found by TLA+ model checking and code review of the SCM pending container
allocation tracking under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Generated with Specula
(Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
