[
https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150530#comment-16150530
]
Thomas Graves edited comment on SPARK-21888 at 9/1/17 1:37 PM:
---
Putting things into SPARK_CONF_DIR will work, the question is more about
convenience for users. In hosted/multitenant environments there is probably a
generic SPARK_CONF_DIR shared by everyone (at least this is how our env works),
for the user to add hbase-site.xml they would have to copy, add files and then
export SPARK_CONF_DIR. If that user continues to use the copied version they
might miss changes to the cluster version, etc. Previously they didn't have to
do this, they just had to set SPARK_CLASSPATH, of course even that doesn't
always work if your cluster env (spark_env.sh) had SPARK_CLASSPATH set in it.
So the question is more of what we think about this for convenience for users.
Personally I think it would be nice to have a config that would allow users to
set an extra classpath on the client side without having to modify the
SPARK_CONF_DIR. thoughts from others?
I think we can move this to an improvement jira, if other people here don't
agree or see the usefulness then we can just close.
was (Author: tgraves):
Putting things into SPARK_CONF_DIR will work, the question is more about
convenience for users. In hosted/multitenant environments there is probably a
generic SPARK_CONF_DIR shared by everyone (at least this is how our env works),
for the user to add hbase-site.xml they would have to copy, add files and then
export SPARK_CONF_DIR. If that user continues to use the copied version they
might miss changes to the cluster version, etc. Previously they didn't have to
do this, they just had to set SPARK_CLASSPATH, of course even that doesn't
always work if your cluster env (spark_env.sh) had SPARK_CLASSPATH set in it.
So the question is more of what we think about this for convenience for users.
Personally I think it would be nice to have a config that would allow users to
set an extra classpath on the client side without having to modify the
SPARK_CONF_DIR?
I think we can move this to an improvement jira, if other people here don't
agree or see the usefulness then we can just close.
> Cannot add stuff to Client Classpath for Yarn Cluster Mode
> --
>
> Key: SPARK-21888
> URL: https://issues.apache.org/jira/browse/SPARK-21888
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.2.0
>Reporter: Parth Gandhi
>Priority: Minor
>
> While running Spark on Yarn in cluster mode, currently there is no way to add
> any config files, jars etc. to Client classpath. An example for this is that
> suppose you want to run an application that uses hbase. Then, unless and
> until we do not copy the necessary config files required by hbase to Spark
> Config folder, we cannot specify or set their exact locations in classpath on
> Client end which we could do so earlier by setting the environment variable
> "SPARK_CLASSPATH".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
