[
https://issues.apache.org/jira/browse/OOZIE-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-1683:
---------------------------------
Attachment: OOZIE-1683.patch
The patch closes the FileSystems opened by any UGIs that the
UserGroupInformationService has on shutdown. It also removes some redundant
code.
No tests, but I verified that the FileSystems are now being closed and removed
from the HDFS FileSystem cache.
> UserGroupInformationService should close any filesystems opened by it
> ---------------------------------------------------------------------
>
> Key: OOZIE-1683
> URL: https://issues.apache.org/jira/browse/OOZIE-1683
> Project: Oozie
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Attachments: OOZIE-1683.patch
>
>
> When you create a FileSystem object in HDFS, it caches it. However, when
> using a UGI to get the FileSystem, it caches it to that object, not its
> “value”. In other words, if you create two UGIs for foo to impersonate bar,
> then they will each create an entry in the cache. In Oozie, we created a
> UserGroupInformationService class that basically sits on top of the HDFS
> cache and caches the UGI objects for impersonating a user so we can reuse the
> same FileSystem object each time. However, when running unit tests, we
> typically destroy and recreate the services for each test (sometimes even
> more) so Oozie’s cache was being lost and we were effectively creating a new
> FileSystem object all the time without closing it.
> Looking at a heap dump after running many of the tests, we saw that there
> were 800+ FileSystem objects open for the same FileSystem.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
