[jira] [Comment Edited] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode
[
https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083081#comment-16083081
]
Saisai Shao edited comment on SPARK-21377 at 7/11/17 10:13 PM:
---
My original purpose is to add jars uploaded by distributed cache to AM
classpath with "$\{PWD\}/*" in AM container setup both in client and cluster
mode, but I guess it may affect the use of "spark.driver.userClassPathFirst" in
cluster mode or contaminate the existing AM classpath, since now jars will also
be existed in AM classpath.
So your suggestion is that we use another context loader to load these jars,
and specifically used for ServiceCredentialProvider, am I right?
was (Author: jerryshao):
My original purpose is to add jars uploaded by distributed cache to AM
classpath with "${PWD}/*" in AM container setup both in client and cluster
mode, but I guess it may affect the use of "spark.driver.userClassPathFirst" in
cluster mode or contaminate the existing AM classpath, since now jars will also
be existed in AM classpath.
So your suggestion is that we use another context loader to load these jars,
and specifically used for ServiceCredentialProvider, am I right?
> Add a new configuration to extend AM classpath in yarn client mode
> --
>
> Key: SPARK-21377
> URL: https://issues.apache.org/jira/browse/SPARK-21377
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.2.0
>Reporter: Yesha Vora
>Priority: Minor
>
> In this issue we have a long running Spark application with secure HBase,
> which requires {{HBaseCredentialProvider}} to get tokens periodically, we
> specify HBase related jars with {{\--packages}}, but these dependencies are
> not added into AM classpath, so when {{HBaseCredentialProvider}} tries to
> initialize HBase connections to get tokens, it will be failed.
> Currently because jars specified with {{\--jars}} or {{\--packages}} are not
> added into AM classpath, the only way to extend AM classpath is to use
> "spark.driver.extraClassPath" which supposed to be used in yarn cluster mode.
> So here we should figure out a solution either to put these dependencies to
> AM classpath or to extend AM classpath with correct configuration.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode
[
https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083060#comment-16083060
]
Saisai Shao edited comment on SPARK-21377 at 7/11/17 9:54 PM:
--
Thanks [~vanzin] for your comment.
Your comment is correct, specifying {{\--packages}} will not add jars to AM, my
original thought is to add main jar and secondary jars automatically into AM
classpath, but this will break the usage of "spark.driver.userClassPathFirst".
So my proposal is to manually specify AM extra classpath with
"spark.yarn.am.extraClassPath" manually, for example specifying HBase classpath
with this configuration. This requires HBase dependencies existed in cluster,
but it may not impact user application's classpath.
was (Author: jerryshao):
Thanks [~vanzin] for your comment.
Your comment is correct, specifying {{\--packages}} will not add jars to AM, my
original thought is to main jar and secondary jars automatically into AM
classpath, but this will break the usage of "spark.driver.userClassPathFirst".
So my proposal is to manually specify AM extra classpath with
"spark.yarn.am.extraClassPath" manually, for example specifying HBase classpath
with this configuration. This requires HBase dependencies existed in cluster,
but it may not impact user application's classpath.
> Add a new configuration to extend AM classpath in yarn client mode
> --
>
> Key: SPARK-21377
> URL: https://issues.apache.org/jira/browse/SPARK-21377
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.2.0
>Reporter: Yesha Vora
>Priority: Minor
>
> In this issue we have a long running Spark application with secure HBase,
> which requires {{HBaseCredentialProvider}} to get tokens periodically, we
> specify HBase related jars with {{\--packages}}, but these dependencies are
> not added into AM classpath, so when {{HBaseCredentialProvider}} tries to
> initialize HBase connections to get tokens, it will be failed.
> Currently because jars specified with {{\--jars}} or {{\--packages}} are not
> added into AM classpath, the only way to extend AM classpath is to use
> "spark.driver.extraClassPath" which supposed to be used in yarn cluster mode.
> So here we should figure out a solution either to put these dependencies to
> AM classpath or to extend AM classpath with correct configuration.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
