[
https://issues.apache.org/jira/browse/MAPREDUCE-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13767396#comment-13767396
]
Chris Nauroth commented on MAPREDUCE-5508:
------------------------------------------
I was +1 for Xi's patch, but I didn't want to commit it too hastily so that the
participants on MAPREDUCE-5351 would get a chance to review. Sandy, thank you
for responding so quickly.
bq. Have you tested this fix?
Yes, Xi repeated his test run of ~200,000 jobs with this patch, and the heap
dump no longer showed leaked instances of {{DistributedFileSystem}}. This is
definitely the source of the leak.
bq. The deeper problem to me is that we are creating a new UGI, which can have
a new subject, which can create a new entry in the FS cache, every time
CleanupQueue#deletePath is called with a null UGI.
Just to clarify, the problem we saw is that a new {{UserGroupInformation}}/new
{{Subject}} gets created regardless of whether or not a null UGI was passed to
{{CleanupQueue#deletePath}}. It's buried behind the last line of this code
snippet:
{code}
protected void deletePath() throws IOException, InterruptedException {
final Path p = getPathForCleanup();
(ugi == null ? UserGroupInformation.getLoginUser() : ugi).doAs(
new PrivilegedExceptionAction<Object>() {
public Object run() throws IOException {
FileSystem fs = p.getFileSystem(conf);
{code}
The last line eventually hits {{FileSystem#Cache#get}}, which calls
{{UserGroupInformation#getCurrentUser}} while constructing the cache key, but
that doesn't always result in the same underlying {{Subject}} instance as the
one that initially created the FS cache entry.
bq. A better fix would be to avoid this, either by having CleanupQueue hold a
UGI of the login user for use in these situations or to avoid the doAs entirely
when the given UGI is null.
We can tell from the heap dump that the leaked instances are associated with
the user UGI and not the login UGI, so I don't think there is a way to use the
login UGI to remove those cache entries. I don't see a way to avoid the doAs,
because we're seeing the leak in the case of the user UGI, so we'd want the
impersonation in place.
Side note: the return value of {{UserGroupInformation#getLoginUser}} is cached
for the lifetime of the process, so it's always going to have the same
{{Subject}} instance. That makes it impossible to have a large, visible leak
in the FS cache for entries associated to the login user.
The only other potential solution I can think of is explicitly passing the
{{FileSystem}} instance to use for the delete and close into
{{PathDeletionContext}}. Then, it wouldn't need to infer the FS to use by
calling {{Path#getFileSystem}} with its implicit creation of a new {{Subject}}.
Do you think something like that would work?
BTW, this problem also made me wonder if it's incorrect for UGI to use an
identity hash code. I couldn't track down the rationale for that. Presumably
it's related to performance. The code of {{Subject#hashCode}} combines data
from a lot of internal data structures, and it all happens while holding the
monitor. This made me think changing the hash code would be too risky.
> JobTracker memory leak caused by unreleased FileSystem objects in
> JobInProgress#cleanupJob
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5508
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5508
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 1-win, 1.2.1
> Reporter: Xi Fang
> Assignee: Xi Fang
> Priority: Critical
> Attachments: MAPREDUCE-5508.patch
>
>
> MAPREDUCE-5351 fixed a memory leak problem but introducing another filesystem
> object (see "tempDirFs") that is not properly released.
> {code} JobInProgress#cleanupJob()
> void cleanupJob() {
> ...
> tempDirFs = jobTempDirPath.getFileSystem(conf);
> CleanupQueue.getInstance().addToQueue(
> new PathDeletionContext(jobTempDirPath, conf, userUGI, jobId));
> ...
> if (tempDirFs != fs) {
> try {
> fs.close();
> } catch (IOException ie) {
> ...
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
