[
https://issues.apache.org/jira/browse/MAPREDUCE-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774280#comment-13774280
]
Xi Fang commented on MAPREDUCE-5508:
------------------------------------
[~cnauroth], thanks for your comments.
bq. Swallowing the InterruptedException is problematic if any upstream code
depends on seeing the thread's interrupted status, so let's restore the
interrupted status in the catch block by calling
Thread.currentThread().interrupt().
If we call Thread.currentThread().interrupt(), is that possible that fs won't
be closed in JobInProgress#cleanupJob()?
bq. If there is an InterruptedException, then we currently would pass a null
tempDirFs to the CleanupQueue, where we'd once again risk leaking memory. I
suggest that if there is an InterruptedException, then we skip adding to the
CleanupQueue and log a warning. This is consistent with the error-handling
strategy in the rest of the method. (It logs warnings.)
I think if the answer to my first question is "fs will be closed in
JobInProgress#cleanupJob()", there will be no memory leak. This is because even
if we pass null into CleanupQueue, the new fs created in
CleanupQueue#deletePath() would be closed anyway.
Thanks Chris.
> JobTracker memory leak caused by unreleased FileSystem objects in
> JobInProgress#cleanupJob
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5508
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5508
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 1-win, 1.2.1
> Reporter: Xi Fang
> Assignee: Xi Fang
> Priority: Critical
> Attachments: MAPREDUCE-5508.1.patch, MAPREDUCE-5508.patch
>
>
> MAPREDUCE-5351 fixed a memory leak problem but introducing another filesystem
> object (see "tempDirFs") that is not properly released.
> {code} JobInProgress#cleanupJob()
> void cleanupJob() {
> ...
> tempDirFs = jobTempDirPath.getFileSystem(conf);
> CleanupQueue.getInstance().addToQueue(
> new PathDeletionContext(jobTempDirPath, conf, userUGI, jobId));
> ...
> if (tempDirFs != fs) {
> try {
> fs.close();
> } catch (IOException ie) {
> ...
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
