[ https://issues.apache.org/jira/browse/YARN-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Szilard Nemeth updated YARN-9519: --------------------------------- Description: The TFile log aggregation file format is not sensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config. In {{LogAggregationTFileController$initInternal}}: {code:java} this.remoteRootLogDir = new Path( conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR)); {code} So the remoteRootLogDir is only aware of the yarn.nodemanager.remote-app-log-dir config, while other file format, like IFile defaults to the file format config, so its priority is higher. >From {{LogAggregationIndexedFileController$initInternal}}: {code:java} String remoteDirStr = String.format( YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT, this.fileControllerName); String remoteDir = conf.get(remoteDirStr); if (remoteDir == null || remoteDir.isEmpty()) { remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR); } {code} (Where these configs are: ) {code:java} public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir"; public static final String NM_REMOTE_APP_LOG_DIR = NM_PREFIX + "remote-app-log-dir"; {code} I suggest TFile should try to obtain the remote dir config from yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not specified falls back to the yarn.nodemanager.remote-app-log-dir config. was: The TFile log aggregation file format is not sensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config. In {{LogAggregationTFileController$initInternal}}: {code:java} this.remoteRootLogDir = new Path( conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR)); {code} So the remoteRootLogDir is only aware of the yarn.nodemanager.remote-app-log-dir config, while other file format, like IFile defaults to the file format config, so its priority is bigger. >From {{LogAggregationIndexedFileController$initInternal}}: {code:java} String remoteDirStr = String.format( YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT, this.fileControllerName); String remoteDir = conf.get(remoteDirStr); if (remoteDir == null || remoteDir.isEmpty()) { remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR); } {code} (Where these configs are: ) {code:java} public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir"; public static final String NM_REMOTE_APP_LOG_DIR = NM_PREFIX + "remote-app-log-dir"; {code} I suggest TFile should try to obtain the remote dir config from yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not specified falls back to the yarn.nodemanager.remote-app-log-dir config. > TFile log aggregation file format is insensitive to the > yarn.log-aggregation.TFile.remote-app-log-dir config > ------------------------------------------------------------------------------------------------------------ > > Key: YARN-9519 > URL: https://issues.apache.org/jira/browse/YARN-9519 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation > Affects Versions: 3.2.0 > Reporter: Adam Antal > Assignee: Adam Antal > Priority: Major > Attachments: YARN-9519.001.patch > > > The TFile log aggregation file format is not sensitive to the > yarn.log-aggregation.TFile.remote-app-log-dir config. > In {{LogAggregationTFileController$initInternal}}: > {code:java} > this.remoteRootLogDir = new Path( > conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, > YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR)); > {code} > So the remoteRootLogDir is only aware of the > yarn.nodemanager.remote-app-log-dir config, while other file format, like > IFile defaults to the file format config, so its priority is higher. > From {{LogAggregationIndexedFileController$initInternal}}: > {code:java} > String remoteDirStr = String.format( > YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT, > this.fileControllerName); > String remoteDir = conf.get(remoteDirStr); > if (remoteDir == null || remoteDir.isEmpty()) { > remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, > YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR); > } > {code} > (Where these configs are: ) > {code:java} > public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT > = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir"; > public static final String NM_REMOTE_APP_LOG_DIR = > NM_PREFIX + "remote-app-log-dir"; > {code} > I suggest TFile should try to obtain the remote dir config from > yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not > specified falls back to the yarn.nodemanager.remote-app-log-dir config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org