[
https://issues.apache.org/jira/browse/HDFS-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287722#comment-13287722
]
Daryn Sharp commented on HDFS-3373:
-----------------------------------
On a related tangent, a similar situation causes the NMs to leak sockets due to
unclosed {{DistributedFileSystems}}. Each one holds a {{DFSClient}} that holds
a cache with at least 1 CLOSE_WAIT socket. The socket gets closed if/when the
{{DFSClient}} looks for a cached socket, but it gets replaced with another
CLOSE_WAIT socket.
With security enabled, a NM leaks 1 socket/user. With security disabled, a NM
will leak *1 socket/job* since the UGI always appears different which causes
multiple instances of the same dfs to pile up in the fs cache.
The UGI/fs-cache is a separate issue, but there should probably be something
like another thread that scans the cached sockets to remove closed ones.
Otherwise {{FileContext}} will be ill-suited for a daemon that needs to create
more than one context since it too will leak sockets.
> FileContext HDFS implementation can leak socket caches
> ------------------------------------------------------
>
> Key: HDFS-3373
> URL: https://issues.apache.org/jira/browse/HDFS-3373
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 2.0.0-alpha
> Reporter: Todd Lipcon
> Assignee: John George
>
> As noted by Nicholas in HDFS-3359, FileContext doesn't have a close() method,
> and thus never calls DFSClient.close(). This means that, until finalizers
> run, DFSClient will hold on to its SocketCache object and potentially have a
> lot of outstanding sockets/fds held on to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
