[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683651#comment-16683651
]
Steve Loughran commented on HADOOP-14556:
-----------------------------------------
Allen,
good q.
# Dtutil only fetches DTs if UGI is in secure mode, whereas fetchdt asks the FS
irrespective of the local security state. Therefore it can issue DTs without
Kerberos. You can't use them for job submission as MR's token fetching (also
used by Distcp) requires Kerberos, as does the spark token collection. But you
can use the tokens collected by fetchdt in other apps, as the [latest relase of
cloudstore
does|https://github.com/steveloughran/cloudstore/releases/tag/tag_2018_11_09b]
# Because the probe for "Are tokens available" doesn't take the FS URI , the
impl has to say "yes" without knowing if the FS actually does.
# Dtutil expects that when a token is requested, the impl always returns 1+
token. Because s3a token issuing is optional (as it is on azure, abfs), if you
ask the FS for a token and it doesn't issue one, you get a stack trace (Array
out of bounds or something similar)
For fetch DT to work in this world, it needs
* service loading to be resilient to classpath problems (FWIW, so does whole
token mechanism: HADOOP-15808)
* FS (or at least s3a FS) code to say "true" whenever probed to see if tokens
are available
* dtutil to be ready to handle the case where "no tokens actually get issued"
(at the very least make it an option)
that means: changes in DTutil, and the fs binding
> S3A to support Delegation Tokens
> --------------------------------
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch,
> HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch,
> HADOOP-14556-018a.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
