[
https://issues.apache.org/jira/browse/HDFS-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He resolved HDFS-17953.
--------------------------------
Fix Version/s: 3.6.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Fix DataNode OOM when pendingIBRs unlimited grows while NameNode is
> unreachable
> -------------------------------------------------------------------------------
>
> Key: HDFS-17953
> URL: https://issues.apache.org/jira/browse/HDFS-17953
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.4.3
> Reporter: yanbin.zhang
> Assignee: yanbin.zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.6.0
>
>
> h2. Problem
> When a DataNode is configured with HA NameNodes
> (\{{dfs.ha.namenodes.<nsId>}}),
> but one of the configured NameNode IDs has no (or a wrong) RPC address — or
> the
> NameNode is permanently unreachable — the
> {\{IncrementalBlockReportManager.pendingIBRs}} map for that actor grows
> without
> bound and eventually causes the DataNode to OOM.
> h2. Root Cause
> In \{{BPOfferService#notifyNamenodeBlock}}, every block receive/delete event
> is
> added to the IBR queue of *every* BPServiceActor, including the actor for the
> unreachable NameNode. In \{{IncrementalBlockReportManager#sendIBRs}}:
> # \{{generateIBRs()}} drains the pending blocks from the queue.
> # \{{namenode.blockReceivedAndDeleted()}} attempts the RPC.
> # on failure, \{{putMissing()}} puts all the blocks *back* into the queue.
> When the NameNode is permanently unreachable:
> * \{{sendIBRs()}} always fails, so blocks are always put back;
> * new block events keep being added via \{{addRDBI()}};
> * \{{pendingIBRs}} grows indefinitely — there is *no size limit, no TTL and no
> eviction*.
> The only existing protection (HDFS-9917) calls \{{clearIBRs()}} during
> {\{reRegister()}} for STANDBY/OBSERVER NNs, but \{{reRegister()}} is never
> triggered when the NN is completely unreachable (no \{{DNA_REGISTER}} command
> can
> be received).
> h2. How to reproduce
> Configure HA with 3 NameNode IDs but only provide RPC addresses for 2:
> {code:xml}
> <property>
> <name>dfs.ha.namenodes.mycluster</name>
> <value>nn1,nn2,nn3</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.mycluster.nn1</name>
> <value>host1:8020</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.mycluster.nn2</name>
> <value>host2:8020</value>
> </property>
> <!-- nn3 RPC address is missing; it falls back to the logical
> nameservice id "mycluster:8020", which never resolves -->
> {code}
> The DataNode creates a BPServiceActor for \{{mycluster:8020}} that can never
> connect. The DataNode log fills with:
> {noformat}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to
> server: mycluster:8020
> {noformat}
> After running for some time the DataNode heap is dominated by
> {\{ReceivedDeletedBlockInfo}} / \{{Block}} / \{{HashMap$Node}} entries
> belonging to
> that dead actor's IBR queue. A production heap dump showed ~223M
> {\{ReceivedDeletedBlockInfo}} instances occupying ~30 GB.
> h2. Impact
> * DataNode OOM / crash.
> * After a restart the heap starts filling again immediately, because the
> configuration is unchanged.
> * Affects any cluster where a NameNode is permanently removed/decommissioned
> without updating \{{dfs.ha.namenodes.<nsId>}}, or where an RPC address is
> simply misconfigured.
> h2. Proposed fix
> # Add a configurable cap on the pending IBR queue,
> {\{dfs.datanode.ibr.max.pending.size}} (default 1,000,000). When exceeded, the
> queue is cleared and a warning is logged.
> # Add a staleness guard \{{dfs.datanode.ibr.max.stale.interval.ms}}
> (default 30 min): if there has been no successful IBR send within this window
> while entries are pending, the queue is cleared.
> # After clearing (overflow or staleness), schedule a Full Block Report so the
> NameNode gets a complete, consistent view once it is reachable again.
> This is safe: the FBR is the ultimate consistency guarantee; IBRs are only an
> optimization for incremental updates between FBRs. Dropping queued IBRs for an
> unreachable NN cannot cause inconsistency because a fresh FBR is sent on
> reconnect/re-register.
> h2. Test plan
> New unit tests in \{{TestIncrementalBlockReportManager}}:
> * \{{testIBRQueueSizeLimit}} — queue is capped and cleared on overflow.
> * \{{testIBRQueueStaleInterval}} — queue is cleared after the stale interval.
> * \{{testIBRQueueNoLimit}} — cap disabled (0) keeps old behavior.
> * \{{testIBRDeduplication}} — per-block dedup still works.
> * \{{testTotalPendingIBRSizeMultipleStorages}} — size counting across
> storages.
> ---
> ## Labels (可选)
> datanode, ibr, memory-leak, oom
> ## 关联关系
> - Relates to / Is related to: **HDFS-9917**, **HDFS-15113**
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
