[jira] [Commented] (YARN-3021) YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp

[Jian He (JIRA)]

    [ 
https://issues.apache.org/jira/browse/YARN-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370885#comment-14370885
 ] 

Jian He commented on YARN-3021:
-------------------------------

thanks for updating !
sorry for being unclear. For the change in DelegationTokenRenewer,  I think we 
only need to perform the check whether the renewer is empty in the for loop at 
line 410: if token renewer is empty, do not add to the tokenList;
I missed one thing that dist cp may also work with webhdfs in which case the 
token kind won't be "HDFS_DELEGATION_TOKEN", so it should not have the 
HDFS_DELEGATION_TOKEN check. please disregard that comment. 
Basically, the for loop may look like something below. 
{code}
    for (Token<?> token : tokens) {
      if (token.isManaged()) {
        if (token.getKind().equals(HDFS_DELEGATION_KIND)) {
          LOG.info(applicationId + " found existing hdfs token " + token);
          hasHdfsToken = true;
        }

        Text renewer = ((Token<AbstractDelegationTokenIdentifier>) token).
            decodeIdentifier().getRenewer();
        if (renewer != null && renewer.toString().equals("")) {
          continue;
        }

        DelegationTokenToRenew dttr = allTokens.get(token);
        if (dttr != null) {
          // If any of the jobs sharing the same token doesn't want to cancel
          // the token, we should not cancel the token.
          if (!evt.shouldCancelAtEnd) {
            dttr.shouldCancelAtEnd = evt.shouldCancelAtEnd;
            LOG.info("Set shouldCancelAtEnd=" + shouldCancelAtEnd
                + " for token " + dttr.token);
          }
          continue;
        }

        tokenList.add(new DelegationTokenToRenew(applicationId, token,
          getConfig(), now, shouldCancelAtEnd, evt.getUser()));
      }
    }
{code}

> YARN's delegation-token handling disallows certain trust setups to operate 
> properly over DistCp
> -----------------------------------------------------------------------------------------------
>
>                 Key: YARN-3021
>                 URL: https://issues.apache.org/jira/browse/YARN-3021
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 2.3.0
>            Reporter: Harsh J
>            Assignee: Yongjun Zhang
>         Attachments: YARN-3021.001.patch, YARN-3021.002.patch, 
> YARN-3021.003.patch, YARN-3021.004.patch, YARN-3021.005.patch, YARN-3021.patch
>
>
> Consider this scenario of 3 realms: A, B and COMMON, where A trusts COMMON, 
> and B trusts COMMON (one way trusts both), and both A and B run HDFS + YARN 
> clusters.
> Now if one logs in with a COMMON credential, and runs a job on A's YARN that 
> needs to access B's HDFS (such as a DistCp), the operation fails in the RM, 
> as it attempts a renewDelegationToken(…) synchronously during application 
> submission (to validate the managed token before it adds it to a scheduler 
> for automatic renewal). The call obviously fails cause B realm will not trust 
> A's credentials (here, the RM's principal is the renewer).
> In the 1.x JobTracker the same call is present, but it is done asynchronously 
> and once the renewal attempt failed we simply ceased to schedule any further 
> attempts of renewals, rather than fail the job immediately.
> We should change the logic such that we attempt the renewal but go easy on 
> the failure and skip the scheduling alone, rather than bubble back an error 
> to the client, failing the app submission. This way the old behaviour is 
> retained.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)