[
https://issues.apache.org/jira/browse/SPARK-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984713#comment-13984713
]
Thomas Graves edited comment on SPARK-1569 at 4/29/14 8:00 PM:
---------------------------------------------------------------
{quote}
@tgravescs ah I see, you're right. I think I assumed incorrectly that the
executor launcher would bundle up the options and send them over, but I don't
actually see that happening anywhere. So this part of the code is actually not
used:
https://github.com/apache/spark/blob/df6d81425bf3b8830988288069f6863de873aee2/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala#L328
What happens is the executor is just getting its configuration from the driver
when the executor launches. And that works in most cases except for security,
which it needs to know about before connecting. Is that right?
{quote}
That is correct. It needs it before connecting. The code you reference handles
add it for the application master but not the executors. it looks like we need
similar code in ExecutorRunnableUtil.prepareCommand.
Yes before we only had SPARK_JAVA_OPTS and that got put as -D on the command
line so it was always set correctly when the executors launched.
was (Author: tgraves):
<quote>
@tgravescs ah I see, you're right. I think I assumed incorrectly that the
executor launcher would bundle up the options and send them over, but I don't
actually see that happening anywhere. So this part of the code is actually not
used:
https://github.com/apache/spark/blob/df6d81425bf3b8830988288069f6863de873aee2/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala#L328
What happens is the executor is just getting its configuration from the driver
when the executor launches. And that works in most cases except for security,
which it needs to know about before connecting. Is that right?
<quote>
That is correct. It needs it before connecting. The code you reference handles
add it for the application master but not the executors. it looks like we need
similar code in ExecutorRunnableUtil.prepareCommand.
Yes before we only had SPARK_JAVA_OPTS and that got put as -D on the command
line so it was always set correctly when the executors launched.
> Spark on Yarn, authentication broken by pr299
> ---------------------------------------------
>
> Key: SPARK-1569
> URL: https://issues.apache.org/jira/browse/SPARK-1569
> Project: Spark
> Issue Type: Sub-task
> Components: YARN
> Affects Versions: 1.0.0
> Reporter: Thomas Graves
> Priority: Blocker
>
> https://github.com/apache/spark/pull/299 changed the way configuration was
> done and passed to the executors. This breaks use of authentication as the
> executor needs to know that authentication is enabled before connecting to
> the driver.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
