Robert Kanter created OOZIE-1683:
------------------------------------
Summary: UserGroupInformationService should close any filesystems
opened by it
Key: OOZIE-1683
URL: https://issues.apache.org/jira/browse/OOZIE-1683
Project: Oozie
Issue Type: Bug
Affects Versions: trunk
Reporter: Robert Kanter
Assignee: Robert Kanter
When you create a FileSystem object in HDFS, it caches it. However, when using
a UGI to get the FileSystem, it caches it to that object, not its “value”. In
other words, if you create two UGIs for foo to impersonate bar, then they will
each create an entry in the cache. In Oozie, we created a
UserGroupInformationService class that basically sits on top of the HDFS cache
and caches the UGI objects for impersonating a user so we can reuse the same
FileSystem object each time. However, when running unit tests, we typically
destroy and recreate the services for each test (sometimes even more) so
Oozie’s cache was being lost and we were effectively creating a new FileSystem
object all the time without closing it.
Looking at a heap dump after running many of the tests, we saw that there were
800+ FileSystem objects open for the same FileSystem.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
