[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-48762832 #560 is merged. Close this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user dbtsai closed the pull request at: https://github.com/apache/spark/pull/987 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45357297 I mean you can set system properties the same way. SparkConf initializes its configuration from system properties, so my patch covers not only your case, but also others (like using a spark-defaults.conf file for spark-submit users). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45363846 Got you. Looking forward to having your patch merged. Thanks. Sent from my Google Nexus 5 On Jun 6, 2014 9:35 AM, Marcelo Vanzin notificati...@github.com wrote: I mean you can set system properties the same way. SparkConf initializes its configuration from system properties, so my patch covers not only your case, but also others (like using a spark-defaults.conf file for spark-submit users). â Reply to this email directly or view it on GitHub https://github.com/apache/spark/pull/987#issuecomment-45357297. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
GitHub user dbtsai opened a pull request: https://github.com/apache/spark/pull/987 [SPARK-1177] Allow SPARK_JAR to be set programmatically in system properties You can merge this pull request into a Git repository by running: $ git pull https://github.com/dbtsai/spark dbtsai-yarn-spark-jar-from-java-property Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/987.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #987 commit 196df1c9fa0c423a30f3b118bf1dd58480cb2fee Author: DB Tsai dbt...@dbtsai.com Date: 2014-05-27T23:07:27Z Allow users to programmatically set the spark jar. commit bdff88ac46bff5aea63e23c24d5d5f00a4e83023 Author: DB Tsai dbt...@dbtsai.com Date: 2014-06-05T22:43:09Z Doc update --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45284869 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45284873 https://github.com/apache/spark/pull/560 has what I believe is a better way of handling this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45285273 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user dbtsai commented on the pull request:
https://github.com/apache/spark/pull/987#issuecomment-45286460
@chesterxgchen
#560 Agree, it's a more throughout way to handle this issue. In the code
you have, it seems that the spark jar setting is moved to conf: SparkConf in
favor of the CONF_SPARK_JAR. But it will make users difficult to set it up
since the Client.scala also has to be changed. Simple question, with your
change, how users can submit job with their own spark jar by passing the
CONF_SPARK_JAR correctly?
def sparkJar(conf: SparkConf) = {
if (conf.contains(CONF_SPARK_JAR)) {
conf.get(CONF_SPARK_JAR)
} else if (System.getenv(ENV_SPARK_JAR) != null) {
logWarning(
s$ENV_SPARK_JAR detected in the system environment. This
variable has been deprecated
sin favor of the $CONF_SPARK_JAR configuration variable.)
System.getenv(ENV_SPARK_JAR)
} else {
SparkContext.jarOfClass(this.getClass).head
}
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45290220 There's no need to change Client.scala with my change; all you need to do is set spark.yarn.jar somewhere - JVM system property, spark-defaults.conf, of in the app's code (modifying the SparkConf instance), and it will be picked up by the Yarn code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45292661 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45292804 The app's code will only run in the application master in yarn-cluster mode, how can yarn client know which jar will be submitted to distributed cache if we set it in the app's spark conf? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45292958 Ok, in cluster mode you can't use SparkConf.set(), but the other two options work fine. You can't do System.setProperty() in cluster mode to achieve that either, so even with your patch, you'd have to use -DSPARK_JAR=foo in the command line for it to work in yarn-cluster mode. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45293874 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/987#issuecomment-45293876 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15489/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1177] Allow SPARK_JAR to be set program...
Github user dbtsai commented on the pull request:
https://github.com/apache/spark/pull/987#issuecomment-45296471
We lunched Spark job inside our tomcat, and we directly use Client.scala
API. With my patch, I can setup the spark jar using System.setProperty() before
val sparkConf = new SparkConf
val args = getArgsFromConf(conf)
new Client(new ClientArguments(args, sparkConf), hadoopConfig,
sparkConf).run
Do you mean that with your work, I can setup the jar location in the
sparkConf which will be passed into the new Client?
Can we have the following in sparkJar method
def sparkJar(conf: SparkConf) = {
if (conf.contains(CONF_SPARK_JAR)) {
conf.get(CONF_SPARK_JAR)
} else if (System.getProperty(ENV_SPARK_JAR) != null) {
logWarning(
s$ENV_SPARK_JAR detected in the system property. This variable
has been deprecated
sin favor of the $CONF_SPARK_JAR configuration variable.)
System.getProperty(ENV_SPARK_JAR)
} else if (System.getenv(ENV_SPARK_JAR) != null) {
logWarning(
s$ENV_SPARK_JAR detected in the system environment. This
variable has been deprecated
sin favor of the $CONF_SPARK_JAR configuration variable.)
System.getenv(ENV_SPARK_JAR)
} else {
SparkContext.jarOfClass(this.getClass).head
}
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
