[
https://issues.apache.org/jira/browse/SPARK-19739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720656#comment-16720656
]
Steve Loughran commented on SPARK-19739:
----------------------------------------
no, leave alone.
* In HADOOP-14556 I'm actually adding that temp one as ahead of the normal
creds in the list, so it will be lifted by default.
* That patch actually adds something way more profound: the ability to create
Delegation Tokens off S3A endpoints, which will be automatic session
credentials, or, with a bit more config, shorter lived role credentials with
restricted access to only the resources you need (specific s3a, any matching
ddb table). With that, you don't need to propagate S3 credentials at all, so
your secrets stay on your local system.
Yes, spark works with this. No, can't do a demo right now, but I'll stick a
video up as soon as I can make one (next week)
> SparkHadoopUtil.appendS3AndSparkHadoopConfigurations to propagate full set of
> AWS env vars
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-19739
> URL: https://issues.apache.org/jira/browse/SPARK-19739
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Steve Loughran
> Assignee: Genmao Yu
> Priority: Minor
> Fix For: 2.2.0
>
>
> {{SparkHadoopUtil.appendS3AndSparkHadoopConfigurations()}} propagates the AWS
> user and secret key to s3n and s3a config options, so getting secrets from
> the user to the cluster, if set.
> AWS also supports session authentication (env var {{AWS_SESSION_TOKEN}}) and
> region endpoints {{AWS_DEFAULT_REGION}}, the latter being critical if you
> want to address V4-auth-only endpoints like frankfurt and Seol.
> These env vars should be picked up and passed down to S3a too. 4+ lines of
> code, though impossible to test unless the existing code is refactored to
> take the env var map[String, String], so allowing a test suite to set the
> values in itds own map.
> side issue: what if only half the env vars are set and users are trying to
> understand why auth is failing? It may be good to build up a string
> identifying which env vars had their value propagate, and log that @ debug,
> while not logging the values, obviously.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
