[jira] [Commented] (YARN-8418) App local logs could leaked if log aggregation fails to initialize for the app

[Suma Shivaprasad (JIRA)]

    [ 
https://issues.apache.org/jira/browse/YARN-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554751#comment-16554751
 ] 

Suma Shivaprasad commented on YARN-8418:
----------------------------------------

Thanks for the clarification [~bibinchundatt]. A couple of more 
questions/comments

1. 
{noformat}
// Schedule the aggregator.
    Runnable aggregatorWrapper = new Runnable() {
      public void run() {
        try {
          appLogAggregator.run();
        } finally {
     *     appLogAggregators.remove(appId);
          closeFileSystems(userUgi);*
        }
      }
    };
{noformat}


If the aggregatorWrapper runnable runs before the enableAppAggregators is 
called for that app during credential updates, then enableAppAggregators will 
not find the app in appLogAggregators map since in finally 
appLogAggregators.remove(appId) is called ;i.e  

{noformat}
 AppLogAggregator aggregator = appLogAggregators.get(appId);
      if (aggregator != null) {
{noformat}

 Am I missing something here?


2. Applications dont get removed from invalidTokenApps except in  
enableAppAggregators. This cause a leak if we never get a credential update 
event for some reason? Should we add logic to remove when application finishes 
i.e APPLICATION_FINISHED is received?

3. Minor nit - In ContainerManagerImp.handleCredentialUpdate, can we rename 
disableApps to disabledApps
{noformat}
Set<ApplicationId> disableApps = logHandler.getDisabledApps();
{noformat}





> App local logs could leaked if log aggregation fails to initialize for the app
> ------------------------------------------------------------------------------
>
>                 Key: YARN-8418
>                 URL: https://issues.apache.org/jira/browse/YARN-8418
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.8.0, 3.0.0-alpha1
>            Reporter: Bibin A Chundatt
>            Assignee: Bibin A Chundatt
>            Priority: Critical
>         Attachments: YARN-8418.001.patch, YARN-8418.002.patch, 
> YARN-8418.003.patch, YARN-8418.004.patch, YARN-8418.005.patch
>
>
> If log aggregation fails init createApp directory container logs could get 
> leaked in NM directory
> For log running application restart of NM after token renewal this case is 
> possible/  Application submission with invalid delegation token



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org