[
https://issues.apache.org/jira/browse/HDDS-16893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16893:
------------------------------
Attachment: TestBugMC1IcrBeforeNodeReport.java
> SCM can grant a datanode's container slot again after the ICR of a new
> container, until the datanode's next node report
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16893
> URL: https://issues.apache.org/jira/browse/HDDS-16893
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-1-send-node-report-with-new-container-icr.patch,
> TestBugMC1IcrBeforeNodeReport.java
>
>
> h3. Mechanism
> SCM counts free container slots on a datanode from two inputs
> ({{PendingContainerTracker}}, {{TwoWindowBucket.checkSpaceAndAdd}}): the
> usable space in the last storage report it processed, and the containers it
> has allocated on that datanode but not yet seen reported. It removes the
> pending entry of a new container when it processes the container's ICR
> ({{IncrementalContainerReportHandler.processICR}}) or first sees it in a full
> container report ({{ContainerReportHandler}}).
> The datanode commits the container's space on its volume when it creates the
> container and sends the ICR at once, in an immediately triggered heartbeat.
> That heartbeat carries no fresh node report. The node report is rebuilt only
> by {{NodeReportPublisher}} every {{hdds.node.report.interval}} (60s by
> default) and stored with {{StateContext.refreshFullReport}}. Until the next
> one, SCM counts the new container neither as pending nor in the reported
> committed space, so it can grant the same slot again.
> The question was raised in review of HDDS-14921 (apache/ozone#10000: "Please
> check if node report is also send in ICR") and not followed up.
> h3. Trigger
> Default configuration, no faults. A datanode creates a container on the first
> write after SCM allocated it, so the pending entry normally lasts only
> milliseconds to seconds. For up to one node report interval afterwards, every
> allocation on that datanode is checked against space that does not include
> the new container. This happens for every new container. It becomes visible
> when the datanode is close enough to full that the extra grants exceed its
> remaining slots.
> h3. Impact
> * SCM allocates containers on datanodes that have no room for them, the over
> allocation the tracker was added to prevent (HDDS-14921, HDDS-15024).
> * The datanode then refuses to create such a container ({{DISK_OUT_OF_SPACE}}
> from the volume choice in {{KeyValueContainer.create}}).
> * On a Ratis pipeline the create runs while the first chunk is written
> ({{ContainerStateMachine.writeStateMachineData}}, then the implicit create in
> {{HddsDispatcher.dispatchRequest}}). {{DISK_OUT_OF_SPACE}} is not one of the
> results {{ContainerStateMachine.handleCommandResult}} tolerates, so the state
> machine is marked unhealthy, the Ratis log failure reaches
> {{XceiverServerRatis.handleNodeLogFailure}} and the pipeline is closed. All
> its open containers close, and in flight writes of other clients on that
> pipeline fail and are retried. For EC the write that triggers the create
> fails and the client retries elsewhere. Read from code, not run end to end.
> * Since HDDS-14925, the datanode accepts creates down to a lower hard limit
> than the spare it reports to SCM. With default settings the gap is 0 on
> volumes up to 1TB, 5GB (one default container) at 1.25TB and 80GB on 16TB, so
> on large volumes a few extra grants still succeed inside that band.
> h3. Reproduction
> [^TestBugMC1IcrBeforeNodeReport.java] ({{hadoop-hdds/server-scm}}) uses real
> SCM components ({{SCMNodeManager}}, {{PipelineManagerImpl}},
> {{ContainerManagerImpl.allocateContainer}},
> {{SCMDatanodeHeartbeatDispatcher}}, {{NodeReportHandler}},
> {{IncrementalContainerReportHandler}}). Three datanodes each have one real
> {{HddsVolume}} sized for one container slot, with their storage reports built
> by {{StorageLocationReport}}. It runs deterministically on unmodified source,
> with the default node report interval and a 1GB container size to keep the
> volumes small.
> * {{icrWithoutNodeReportGrantsSlotTwice}} plays the current datanode: create,
> then a heartbeat with only the ICR. A second allocation while the first is
> pending is refused. After the ICR it succeeds on the same full pipeline, and
> the datanode's volume choice for that container throws
> {{DiskOutOfSpaceException}}. The test asserts the correct behaviour and
> fails: {{[second container granted on datanodes with room for one] expected:
> null but was: ContainerInfo\{id=#3, state=OPEN, ...\}}}.
> * {{icrWithNodeReportRefusesSecondSlot}} sends the node report in the same
> heartbeat as the ICR and passes: SCM refuses the second allocation.
> h3. Patch
> [^MC-1-send-node-report-with-new-container-icr.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 6818707698c.
> When the datanode's ICR sender ({{OzoneContainer}}) sends the ICR of an OPEN
> replica, which only happens after a container create, it first refreshes the
> node report with {{getNodeReport()}}. The heartbeat that carries the ICR then
> normally also carries a node report that includes the committed space, so SCM
> sees that space no later than, or shortly after, it releases the pending
> entry on the ICR. SCM handles {{NODE_REPORT}} and ICRs on different
> executors, so when the node report is processed first the container is
> briefly counted twice, which only delays allocations. SCM code is unchanged.
> Cost: on the first write of each new container, every replica builds a node
> report, which walks the datanode's whole container set
> ({{ContainerSet.getOpenContainerCountsByVolume}}) and takes the volume set
> read lock ({{MutableVolumeSet.getStorageReport}}), inside the ICR sender's
> {{synchronized (containerSet)}}. That is linear in the number of containers
> on the datanode, once per create. The extra report goes in the heartbeat the
> ICR already triggers, so there is no extra RPC; SCM and Recon process one
> more node report per create.
> Remaining windows: the heartbeat thread reads the full reports and the
> incremental reports separately
> ({{StateContext.getAllAvailableReportsUpToLimit}}), so the ICR can
> occasionally go out one heartbeat before the node report; a periodic
> {{NodeReportPublisher}} run that built its report before the create can
> overwrite the fresh one; SCM may process the ICR before the node report of
> the same heartbeat; and a heartbeat that fails before SCM processes it is
> resent with the ICR but without the node report. The same gap for replication
> imports ({{SCMNodeManager.opCompleted}} on the import ICR) is not addressed
> here.
> An SCM side alternative would keep the pending entry after the ICR until SCM
> has processed a node report from a later heartbeat than the one that carried
> the ICR. That is nearly sound and would also cover older datanodes,
> replication imports and the resend after a failed heartbeat. It needs more
> per datanode state, the heartbeat order carried to the separate report
> executors, and it counts each new container twice for up to one node report
> interval. The datanode change is the smaller first step; the two do not
> exclude each other.
> Covered by {{testNewContainerReportSentWithNodeReport}} in the existing
> {{TestOzoneContainer}}. It checks that the node report reaches
> {{StateContext}} before the ICR and contains the container's committed bytes,
> and that later ICRs of the container do not refresh it. Without the change it
> fails for all 8 schema and layout cases. With it, the test and the
> container-service suites around the report path ({{TestHddsDispatcher}},
> {{TestStateContext}}, {{TestReportPublisher}}, {{TestHeartbeatEndpointTask}},
> {{TestKeyValueHandler}}, {{TestKeyValueHandlerWithUnhealthyContainer}},
> {{TestContainerPersistence}}, {{TestCloseContainerCommandHandler}},
> {{TestDeleteContainerCommandHandler}}) pass. Checkstyle is clean. Two tests
> that bind the default port 9886 could not run on the test machine, with or
> without the change.
> Found by TLA+ model checking and code review of the SCM pending container
> allocation tracking under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
