[
https://issues.apache.org/jira/browse/HDFS-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412916#comment-13412916
]
Daryn Sharp commented on HDFS-3646:
-----------------------------------
bq. IMO, this will a leak from application side, since there is a bug in
closing the streams from app.
Agreed, but it can have pretty severe consequences that aren't easily detected
unless explicitly hunting for leaks. It makes me uneasy that an out of scope
fs stream can cause a massive leak of heavy objects, threads, and tie up
sockets that may exhaust fds and/or memory for long running processes.
Emitting an angry log error for lost & unclosed streams may be more beneficial.
I don't think a finalizer on the fs will work. If I do {{in =
path.getFileSystem(conf).open(...)}}, the fs might get garbage collected but we
certainly don't want its finalizer to shoot the dfs client that is still
holding open a stream. Maybe a finalizer on the dfs client, but in any case,
the circular hard references need to be broken somehow.
> LeaseRenewer can hold reference to inactive DFSClient instances forever
> -----------------------------------------------------------------------
>
> Key: HDFS-3646
> URL: https://issues.apache.org/jira/browse/HDFS-3646
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Priority: Critical
> Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
>
>
> If {{LeaseRenewer#closeClient()}} is not called, {{LeaseRenewer}} keeps the
> reference to a {{DFSClient}} instance in {{dfsclients}} forever. This
> prevents {{DFSClient}}, {{LeaseRenewer}}, conf, etc. from being garbage
> collected, leading to memory leak.
> {{LeaseRenewer}} should remove the reference after some delay, if a
> {{DFSClient}} instance no longer has active streams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
