[
https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905247#comment-15905247
]
Thomas Graves edited comment on SPARK-19143 at 3/10/17 3:20 PM:
----------------------------------------------------------------
Made some comments in the design doc.
My original idea was to add rpc between client and driver so it works in both
yarn client and cluster mode. that rpc can be used for other things as well
(like a -kill command or get status straight from driver). For yarn side the
credentials are there and available from RM for a client to talk to driver, we
just have to make that token work with our rpc which I don't think should not
be to difficult just needs to be done. we can continue to support the
--keytab/--principal arguments and have the AM update the tokens. It would be
best to update that to push to executors via the rpc too.
was (Author: tgraves):
Made some comments in the design doc.
My original idea was to add rpc between client and driver so it works in both
yarn client and cluster mode. that rpc can be used for other things as well
(like a -kill command or get status straight from driver. For yarn side the
credentials are there and available from RM for a client to talk to driver, we
just have to make that token work with our rpc which I don't think should not
be to difficult just needs to be done. we can continue to support the
--keytab/--principal arguments and have the AM update the tokens. It would be
best to update that to push to executors via the rpc too.
> API in Spark for distributing new delegation tokens (Improve delegation token
> handling in secure clusters)
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19143
> URL: https://issues.apache.org/jira/browse/SPARK-19143
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 2.0.2, 2.1.0
> Reporter: Ruslan Dautkhanov
>
> Spin off from SPARK-14743 and comments chain in [recent comments|
> https://issues.apache.org/jira/browse/SPARK-5493?focusedCommentId=15802179&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15802179]
> in SPARK-5493.
> Spark currently doesn't have a way for distribution new delegation tokens.
> Quoting [~vanzin] from SPARK-5493
> {quote}
> IIRC Livy doesn't yet support delegation token renewal. Once it reaches the
> TTL, the session is unusable.
> There might be ways to hack support for that without changes in Spark, but
> I'd like to see a proper API in Spark for distributing new delegation tokens.
> I mentioned that in SPARK-14743, but although that bug is closed, that
> particular feature hasn't been implemented yet.
> {quote}
> Other thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
