[
https://issues.apache.org/jira/browse/MAPREDUCE-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13767698#comment-13767698
]
Chris Nauroth commented on MAPREDUCE-5508:
------------------------------------------
bq. If I understand correctly, in that case the same UGI instance
(JobInProgress.userUGI) that was used to create the fs is used to close it, so
having different subjects is not possible.
The 2 UGI instances (one passed in to {{PathDeletionContext}} and one created
implicitly by the call to {{Path#getFileSystem}}) do have the same value, but
they have different identities. Even though they are logically equivalent,
there are 2 different underlying {{Subject}} instances with different identity
hash codes. Thus, the cache entry used during creation of the {{FileSystem}}
will be different from the one used during close, which causes the leak.
Another way of saying this is that even though the same {{Principal}} is used,
{{Path#getFileSystem}} will create a different {{Subject}}, because
{{CleanupQueue}} is running inside a different JAAS {{AccessControlContext}}.
The {{Subject}} is a function of not only the {{Principal}} but also the
{{AccessControlContext}}. (Again, both the {{AccessControlContext}} and
{{Subject}} may be logically equivalent, but we're observing that they are not
the same instances, so they have different identity hash codes.)
If you really want to avoid the extra close, then the only other possible
solution that I can think of would involve passing the {{FileSystem}} instance
to use into the {{PathDeletionContext}}. This would make it explicit instead
of relying on the implicit cache lookup of {{Path#getFileSystem}}. I haven't
completely thought through if that approach would have other side effects
though.
> JobTracker memory leak caused by unreleased FileSystem objects in
> JobInProgress#cleanupJob
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5508
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5508
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 1-win, 1.2.1
> Reporter: Xi Fang
> Assignee: Xi Fang
> Priority: Critical
> Attachments: MAPREDUCE-5508.patch
>
>
> MAPREDUCE-5351 fixed a memory leak problem but introducing another filesystem
> object (see "tempDirFs") that is not properly released.
> {code} JobInProgress#cleanupJob()
> void cleanupJob() {
> ...
> tempDirFs = jobTempDirPath.getFileSystem(conf);
> CleanupQueue.getInstance().addToQueue(
> new PathDeletionContext(jobTempDirPath, conf, userUGI, jobId));
> ...
> if (tempDirFs != fs) {
> try {
> fs.close();
> } catch (IOException ie) {
> ...
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
