Siyao Meng created HDDS-16893:
---------------------------------
Summary: SCM can grant a datanode's container slot again after the
ICR of a new container, until the datanode's next node report
Key: HDDS-16893
URL: https://issues.apache.org/jira/browse/HDDS-16893
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-1-send-node-report-with-new-container-icr.patch,
TestBugMC1IcrBeforeNodeReport.java
h3. Mechanism
SCM counts free container slots on a datanode from two inputs
({{PendingContainerTracker}}, {{TwoWindowBucket.checkSpaceAndAdd}}): the usable
space in the last storage report it processed, and the containers it has
allocated on that datanode but not yet seen reported. It removes the pending
entry of a new container when it processes the container's ICR
({{IncrementalContainerReportHandler.processICR}}) or first sees it in a full
container report ({{ContainerReportHandler}}).
The datanode commits the container's space on its volume when it creates the
container and sends the ICR at once, in an immediately triggered heartbeat.
That heartbeat carries no fresh node report. The node report is rebuilt only by
{{NodeReportPublisher}} every {{hdds.node.report.interval}} (60s by default)
and stored with {{StateContext.refreshFullReport}}. Until the next one, SCM
counts the new container neither as pending nor in the reported committed
space, so it can grant the same slot again.
The question was raised in review of HDDS-14921 (apache/ozone#10000: "Please
check if node report is also send in ICR") and not followed up.
h3. Trigger
Default configuration, no faults. A datanode creates a container on the first
write after SCM allocated it, so the pending entry normally lasts only
milliseconds to seconds. For up to one node report interval afterwards, every
allocation on that datanode is checked against space that does not include the
new container. This happens for every new container. It becomes visible when
the datanode is close enough to full that the extra grants exceed its remaining
slots.
h3. Impact
* SCM allocates containers on datanodes that have no room for them, the over
allocation the tracker was added to prevent (HDDS-14921, HDDS-15024).
* The datanode then refuses to create such a container ({{DISK_OUT_OF_SPACE}}
from the volume choice in {{KeyValueContainer.create}}).
* On a Ratis pipeline the create runs while the first chunk is written
({{ContainerStateMachine.writeStateMachineData}}, then the implicit create in
{{HddsDispatcher.dispatchRequest}}). {{DISK_OUT_OF_SPACE}} is not one of the
results {{ContainerStateMachine.handleCommandResult}} tolerates, so the state
machine is marked unhealthy, the Ratis log failure reaches
{{XceiverServerRatis.handleNodeLogFailure}} and the pipeline is closed. All its
open containers close, and in flight writes of other clients on that pipeline
fail and are retried. For EC the write that triggers the create fails and the
client retries elsewhere. Read from code, not run end to end.
* Since HDDS-14925, the datanode accepts creates down to a lower hard limit
than the spare it reports to SCM. With default settings the gap is 0 on volumes
up to 1TB, 5GB (one default container) at 1.25TB and 80GB on 16TB, so on large
volumes a few extra grants still succeed inside that band.
h3. Reproduction
[^TestBugMC1IcrBeforeNodeReport.java] ({{hadoop-hdds/server-scm}}) uses real
SCM components ({{SCMNodeManager}}, {{PipelineManagerImpl}},
{{ContainerManagerImpl.allocateContainer}}, {{SCMDatanodeHeartbeatDispatcher}},
{{NodeReportHandler}}, {{IncrementalContainerReportHandler}}). Three datanodes
each have one real {{HddsVolume}} sized for one container slot, with their
storage reports built by {{StorageLocationReport}}. It runs deterministically
on unmodified source, with the default node report interval and a 1GB container
size to keep the volumes small.
* {{icrWithoutNodeReportGrantsSlotTwice}} plays the current datanode: create,
then a heartbeat with only the ICR. A second allocation while the first is
pending is refused. After the ICR it succeeds on the same full pipeline, and
the datanode's volume choice for that container throws
{{DiskOutOfSpaceException}}. The test asserts the correct behaviour and fails:
{{[second container granted on datanodes with room for one] expected: null but
was: ContainerInfo\{id=#3, state=OPEN, ...\}}}.
* {{icrWithNodeReportRefusesSecondSlot}} sends the node report in the same
heartbeat as the ICR and passes: SCM refuses the second allocation.
h3. Patch
[^MC-1-send-node-report-with-new-container-icr.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
6818707698c.
When the datanode's ICR sender ({{OzoneContainer}}) sends the ICR of an OPEN
replica, which only happens after a container create, it first refreshes the
node report with {{getNodeReport()}}. The heartbeat that carries the ICR then
normally also carries a node report that includes the committed space, so SCM
sees that space no later than, or shortly after, it releases the pending entry
on the ICR. SCM handles {{NODE_REPORT}} and ICRs on different executors, so
when the node report is processed first the container is briefly counted twice,
which only delays allocations. SCM code is unchanged.
Cost: on the first write of each new container, every replica builds a node
report, which walks the datanode's whole container set
({{ContainerSet.getOpenContainerCountsByVolume}}) and takes the volume set read
lock ({{MutableVolumeSet.getStorageReport}}), inside the ICR sender's
{{synchronized (containerSet)}}. That is linear in the number of containers on
the datanode, once per create. The extra report goes in the heartbeat the ICR
already triggers, so there is no extra RPC; SCM and Recon process one more node
report per create.
Remaining windows: the heartbeat thread reads the full reports and the
incremental reports separately
({{StateContext.getAllAvailableReportsUpToLimit}}), so the ICR can occasionally
go out one heartbeat before the node report; a periodic {{NodeReportPublisher}}
run that built its report before the create can overwrite the fresh one; SCM
may process the ICR before the node report of the same heartbeat; and a
heartbeat that fails before SCM processes it is resent with the ICR but without
the node report. The same gap for replication imports
({{SCMNodeManager.opCompleted}} on the import ICR) is not addressed here.
An SCM side alternative would keep the pending entry after the ICR until SCM
has processed a node report from a later heartbeat than the one that carried
the ICR. That is nearly sound and would also cover older datanodes, replication
imports and the resend after a failed heartbeat. It needs more per datanode
state, the heartbeat order carried to the separate report executors, and it
counts each new container twice for up to one node report interval. The
datanode change is the smaller first step; the two do not exclude each other.
Covered by {{testNewContainerReportSentWithNodeReport}} in the existing
{{TestOzoneContainer}}. It checks that the node report reaches {{StateContext}}
before the ICR and contains the container's committed bytes, and that later
ICRs of the container do not refresh it. Without the change it fails for all 8
schema and layout cases. With it, the test and the container-service suites
around the report path ({{TestHddsDispatcher}}, {{TestStateContext}},
{{TestReportPublisher}}, {{TestHeartbeatEndpointTask}},
{{TestKeyValueHandler}}, {{TestKeyValueHandlerWithUnhealthyContainer}},
{{TestContainerPersistence}}, {{TestCloseContainerCommandHandler}},
{{TestDeleteContainerCommandHandler}}) pass. Checkstyle is clean. Two tests
that bind the default port 9886 could not run on the test machine, with or
without the change.
Found by TLA+ model checking and code review of the SCM pending container
allocation tracking under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
