[
https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815078#comment-15815078
]
Thomas Graves commented on SPARK-19143:
---------------------------------------
[~mridulm80] You say "we added", you are saying you have already implemented
this?
It would definitely be nice to support pushing tokens from say a gateway so you
don't have to ship the keytab because shipping the keytab is much less secure.
It would also be nice to not use HDFS to store and transfer the tokens. My
initial thought was to create an rpc between the client on gateways and the
driver/AM (running on yarn node) and transfer the new tokens that way. ideally
it would also then be transferred to the executors via rpc vs again storing in
hdfs. All that would be more secure then storing in hdfs.
We could add a command to spark-submit to get and push new credentials. For
long running jobs it would have to happen periodically (< every 24 hours) but
for initial that could be done via cron or other mechanism.
> API in Spark for distributing new delegation tokens (Improve delegation token
> handling in secure clusters)
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19143
> URL: https://issues.apache.org/jira/browse/SPARK-19143
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 2.0.2, 2.1.0
> Reporter: Ruslan Dautkhanov
>
> Spin off from SPARK-14743 and comments chain in [recent comments|
> https://issues.apache.org/jira/browse/SPARK-5493?focusedCommentId=15802179&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15802179]
> in SPARK-5493.
> Spark currently doesn't have a way for distribution new delegation tokens.
> Quoting [~vanzin] from SPARK-5493
> {quote}
> IIRC Livy doesn't yet support delegation token renewal. Once it reaches the
> TTL, the session is unusable.
> There might be ways to hack support for that without changes in Spark, but
> I'd like to see a proper API in Spark for distributing new delegation tokens.
> I mentioned that in SPARK-14743, but although that bug is closed, that
> particular feature hasn't been implemented yet.
> {quote}
> Other thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
