[
https://issues.apache.org/jira/browse/SPARK-9384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14643127#comment-14643127
]
Mark Grover edited comment on SPARK-9384 at 7/27/15 6:15 PM:
-------------------------------------------------------------
Personally, there are two easy ways to solve this problem:
1. To apply the driver's classpath to executor, if executor classpath is not
set.
2. To create a new property, something like 'spark.common.classpath' that gets
applied to both the driver and executor classpath, in addition to the specific
{code}
spark.{driver|executor}.classpath properties
{code}
In general, I agree with the sentiment that having too many properties can be a
mess but in this case, #1 can lead to too drastic of a behavior change on an
upgrade, so I am strongly leaning towards #2. That would be backwards
compatible in the sense, that if you don't use the new property nothing
changes, but for new users and for those who want to make their configuration
management easier, they would just need to set things in only one place.
I have taken a quick pass at it
[here|https://github.com/markgrover/spark/compare/pmaster...classpath2], I'd
appreciate if someone could take a quick look and tell me if I am on the right
track. No pull request just yet because I want to convince myself it works by
testing it more, before I do so:-)
Your thoughts and suggestions are greatly appreciated!
was (Author: mgrover):
Personally, there are two easy ways to solve this problem:
1. To apply the driver's classpath to executor, if executor classpath is not
set.
2. To create a new property, something like 'spark.common.classpath' that gets
applied to both the driver and executor classpath, in addition to the specific
spark.{driver|executor}.classpath properties.
In general, I agree with the sentiment that having too many properties can be a
mess but in this case, #1 can lead to too drastic of a behavior change on an
upgrade, so I am strongly leaning towards #2. That would be backwards
compatible in the sense, that if you don't use the new property nothing
changes, but for new users and for those who want to make their configuration
management easier, they would just need to set things in only one place.
I have taken a quick pass at it
[here|https://github.com/markgrover/spark/compare/pmaster...classpath2], I'd
appreciate if someone could take a quick look and tell me if I am on the right
track. No pull request just yet because I want to convince myself it works by
testing it more, before I do so:-)
Your thoughts and suggestions are greatly appreciated!
> Easier setting of executor and driver classpath
> -----------------------------------------------
>
> Key: SPARK-9384
> URL: https://issues.apache.org/jira/browse/SPARK-9384
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 1.4.1
> Reporter: Mark Grover
> Priority: Minor
>
> Currently one can set extra jars in the driver classpath using
> {{spark.driver.extraClassPath}} property. Similarly, the executor's extra
> classpath can be set using {{spark.executor.extraClassPath}} property.
> However, many users (myself) included have often suffered from the dogma of
> setting one and missing the other. Causing a lot of wasted time. And, this
> JIRA is an effort to see if the status quo can be made better.
> More details in comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
