[ https://issues.apache.org/jira/browse/YARN-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552327#comment-16552327 ]
Bibin A Chundatt commented on YARN-8418: ---------------------------------------- {quote}1) Is it possible to send the "new token arrived" message to LogAggregationService instead of handling inside ContainersManagerImpl? {quote} Currently as per RM delegation token renewer, all rm system credentials(all apps) are send to all the node manager once renewed. To optimize number of events, added logic in Container Manager based on disabled apps. Will update event to token arrived enabled creates confusion. {quote}Following logic: AppLogAggregator aggregator = appLogAggregators.get(appId); {quote} To handle possible race condition added the same. Lets say application finish -> aggregators removed -> then event arrives for renewed apps rare case. {quote}5) What happens if token arrives after AppLogAggregator removed from context? Is it possible? If yes, are we going to remove log dir for this case? {quote} Yes. We can have a follow up jira for this case.Due to xy issues log aggregation fails currently we delete the logs immediately . We could add an option for delayed delete for aggregation failure cases. {quote}6) Have u done tests in real cluster to prove it work? Just to make sure we're pushing the right fix to 3.1.1 given we don't have much time before RC. {quote} Did the test on real cluster for patch-004 and working fine. *Configuration* delegationTokenMaxLifetime default value/336 delegationTokenRenewInterval =default value/336 delegationKeyUpdateInterval= default value/336 yarn.resourcemanager.system-credentials.valid-time-remaining=default value/336 Configure above confs in all services. Submit application Restart NM after max life time of token. Make sure Apps finish after expiry time. Will be able to reproduce in 30 mins . > App local logs could leaked if log aggregation fails to initialize for the app > ------------------------------------------------------------------------------ > > Key: YARN-8418 > URL: https://issues.apache.org/jira/browse/YARN-8418 > Project: Hadoop YARN > Issue Type: Bug > Affects Versions: 2.8.0, 3.0.0-alpha1 > Reporter: Bibin A Chundatt > Assignee: Bibin A Chundatt > Priority: Critical > Attachments: YARN-8418.001.patch, YARN-8418.002.patch, > YARN-8418.003.patch, YARN-8418.004.patch > > > If log aggregation fails init createApp directory container logs could get > leaked in NM directory > For log running application restart of NM after token renewal this case is > possible/ Application submission with invalid delegation token -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org