yanbin.zhang created HDFS-17953:
-----------------------------------
Summary: DataNode IBR pendingIBRs grows unbounded when a NameNode
is unreachable, causing OOM
Key: HDFS-17953
URL: https://issues.apache.org/jira/browse/HDFS-17953
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.4.3
Reporter: yanbin.zhang
Assignee: yanbin.zhang
h2. Problem
When a DataNode is configured with HA NameNodes (\{{dfs.ha.namenodes.<nsId>}}),
but one of the configured NameNode IDs has no (or a wrong) RPC address — or the
NameNode is permanently unreachable — the
{\{IncrementalBlockReportManager.pendingIBRs}} map for that actor grows without
bound and eventually causes the DataNode to OOM.
h2. Root Cause
In \{{BPOfferService#notifyNamenodeBlock}}, every block receive/delete event is
added to the IBR queue of *every* BPServiceActor, including the actor for the
unreachable NameNode. In \{{IncrementalBlockReportManager#sendIBRs}}:
# \{{generateIBRs()}} drains the pending blocks from the queue.
# \{{namenode.blockReceivedAndDeleted()}} attempts the RPC.
# on failure, \{{putMissing()}} puts all the blocks *back* into the queue.
When the NameNode is permanently unreachable:
* \{{sendIBRs()}} always fails, so blocks are always put back;
* new block events keep being added via \{{addRDBI()}};
* \{{pendingIBRs}} grows indefinitely — there is *no size limit, no TTL and no
eviction*.
The only existing protection (HDFS-9917) calls \{{clearIBRs()}} during
{\{reRegister()}} for STANDBY/OBSERVER NNs, but \{{reRegister()}} is never
triggered when the NN is completely unreachable (no \{{DNA_REGISTER}} command
can
be received).
h2. How to reproduce
Configure HA with 3 NameNode IDs but only provide RPC addresses for 2:
{code:xml}
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2,nn3</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>host1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>host2:8020</value>
</property>
<!-- nn3 RPC address is missing; it falls back to the logical
nameservice id "mycluster:8020", which never resolves -->
{code}
The DataNode creates a BPServiceActor for \{{mycluster:8020}} that can never
connect. The DataNode log fills with:
{noformat}
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to
server: mycluster:8020
{noformat}
After running for some time the DataNode heap is dominated by
{\{ReceivedDeletedBlockInfo}} / \{{Block}} / \{{HashMap$Node}} entries
belonging to
that dead actor's IBR queue. A production heap dump showed ~223M
{\{ReceivedDeletedBlockInfo}} instances occupying ~30 GB.
h2. Impact
* DataNode OOM / crash.
* After a restart the heap starts filling again immediately, because the
configuration is unchanged.
* Affects any cluster where a NameNode is permanently removed/decommissioned
without updating \{{dfs.ha.namenodes.<nsId>}}, or where an RPC address is
simply misconfigured.
h2. Proposed fix
# Add a configurable cap on the pending IBR queue,
{\{dfs.datanode.ibr.max.pending.size}} (default 1,000,000). When exceeded, the
queue is cleared and a warning is logged.
# Add a staleness guard \{{dfs.datanode.ibr.max.stale.interval.ms}}
(default 30 min): if there has been no successful IBR send within this window
while entries are pending, the queue is cleared.
# After clearing (overflow or staleness), schedule a Full Block Report so the
NameNode gets a complete, consistent view once it is reachable again.
This is safe: the FBR is the ultimate consistency guarantee; IBRs are only an
optimization for incremental updates between FBRs. Dropping queued IBRs for an
unreachable NN cannot cause inconsistency because a fresh FBR is sent on
reconnect/re-register.
h2. Test plan
New unit tests in \{{TestIncrementalBlockReportManager}}:
* \{{testIBRQueueSizeLimit}} — queue is capped and cleared on overflow.
* \{{testIBRQueueStaleInterval}} — queue is cleared after the stale interval.
* \{{testIBRQueueNoLimit}} — cap disabled (0) keeps old behavior.
* \{{testIBRDeduplication}} — per-block dedup still works.
* \{{testTotalPendingIBRSizeMultipleStorages}} — size counting across storages.
---
## Labels (可选)
datanode, ibr, memory-leak, oom
## 关联关系
- Relates to / Is related to: **HDFS-9917**, **HDFS-15113**
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
