Rajesh Balamohan created YARN-11039:
---------------------------------------
Summary:
LogAggregationFileControllerFactory::getFileControllerForRead should close FS
Key: YARN-11039
URL: https://issues.apache.org/jira/browse/YARN-11039
Project: Hadoop YARN
Issue Type: Improvement
Components: log-aggregation
Reporter: Rajesh Balamohan
getFileControllerForRead::getFileControllerForRead internally opens up a new FS
object everytime and is not closed.
When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
Lines of interest:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
{noformat}
try {
Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
appOwner);
if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
appOwner).hasNext()) {
return fileController;
}
} catch (Exception ex) {
diagnosticsMsg.append(ex.getMessage() + "\n");
continue;
}
{noformat}
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
{noformat}
public static RemoteIterator<FileStatus> getNodeFiles(Configuration conf,
Path remoteAppLogDir, ApplicationId appId, String appOwner)
throws IOException {
Path qualifiedLogDir =
FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
return FileContext.getFileContext(
qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
}
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
