YUBI LEE created HDFS-17965:
-------------------------------
Summary: DFSInputStream.close() leaks the block reader socket when
the DFSClient is already closed
Key: HDFS-17965
URL: https://issues.apache.org/jira/browse/HDFS-17965
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Environment: DFSInputStream.close() calls dfsClient.checkOpen() before
closeCurrentBlockReaders():
{code:java}
public synchronized void close() throws IOException {
try {
if (!closed.compareAndSet(false, true)) { ... return; }
dfsClient.checkOpen(); // throws if the client is closed
...
closeCurrentBlockReaders(); // never reached in that case
super.close();
{code}
If the DFSClient has already been closed (for example via
FileSystem.closeAllForUGI() during shutdown/cleanup), checkOpen() throws
"Filesystem closed" and the current block reader is never closed. Because
the closed flag has already been CAS-ed(Compare-And-Set) to true, any further
close() call
is a no-op, so the block reader's socket to the DataNode can never be
released for the lifetime of the client JVM.
Production impact we observed: with DefaultContainerExecutor, the YARN
NodeManager runs ContainerLocalizer inside the NM JVM. When a container is
killed while localizing (YARN-11856), runLocalization()'s finally block
closes the UGI filesystems while the download thread still holds an open
DFSInputStream. The subsequent stream close() aborts at checkOpen() and
leaks the socket inside the long-lived NM process. On the DataNode side the
connection is stuck in FIN_WAIT1 with a send queue that never drains; on a
~600-node cluster this accumulated continuously until DataNodes carried
large numbers of FIN_WAIT1 connections and NodeManagers leaked fds.
Fix: release the block reader in a finally block so close() always frees
the socket, while preserving the existing "Filesystem closed" exception
behavior.
Reporter: YUBI LEE
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
