[
https://issues.apache.org/jira/browse/YARN-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396044#comment-15396044
]
Jason Lowe commented on YARN-5438:
----------------------------------
Thanks for the patch, Rohith! This probably works for the HiveServer2 case iff
the server never tries to use the filesystem after the timeline client is
closed. However the timeline client is not just used by HS2, and I think this
patch will be problematic for any code that could still use the filesystem
after the timeline client is closed. Since the filesystem cache will
implicitly link what looks like two separate creations of a filesystem to a
single instance, closing one will break any subsequent use of the other.
This makes me think HS2 is missing a closeAllforUGI call in it somewhere to
make sure when it's done for a certain user it cleans up all the filesystems
associated with that user. It also makes me wonder why we haven't implemented
a reference-counting cache for the filesystem by now.
> TimelineClientImpl leaking FileSystem Instances causing Long running services
> like HiverServer2 daemon going OOM
> ----------------------------------------------------------------------------------------------------------------
>
> Key: YARN-5438
> URL: https://issues.apache.org/jira/browse/YARN-5438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
> Affects Versions: 2.8.0, 2.7.3
> Reporter: Karam Singh
> Assignee: Rohith Sharma K S
> Attachments: YARN-5438.0.patch
>
>
> TimelineClientImpl leaking FileSystem Instances causing Long running services
> like HiverServer2 daemon going OOM
> In org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl,
> FileSystem.newInstance is invoked and is not closed. Causing over time
> Filesystem instances getting accumulated in long runninh Client (like
> Hiveserver2), finally causing them to OOM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
