Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-152407168
@msannell, good catch. Thanks! I will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pro
Github user msannell commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-152298340
I believe I've noticed a problem with this fix. You fixed
core/src/main/scala/org/apache/spark/deploy/RRunner.scala to handle
"spark.r.command" (with nice suppor
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9179
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150753284
Thanks @sun-rui -- LGTM. Merging this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project do
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150747915
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150747916
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150747895
**[Test build #44277 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44277/consoleFull)**
for PR 9179 at commit
[`4f44138`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150731510
**[Test build #44277 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44277/consoleFull)**
for PR 9179 at commit
[`4f44138`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150730660
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150730649
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150729239
@shivaram, good idea, agree.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150311495
+1 on `spark.r.driver.command` and `spark.r.command`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150280544
Hmm lets use `spark.r.driver.command` and `spark.r.command`. Its better to
use `command` than `rscript`. For the backwards compatibility thing can we just
support both
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150137114
@shivaram, some thought on the naming consistency of options.
For pySpark, the name convention is spark.python.xxx, for example,
spark.python.profile
And we a doc
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150127390
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150127393
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150127206
**[Test build #44130 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44130/consoleFull)**
for PR 9179 at commit
[`bfd42e1`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150108576
**[Test build #44130 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44130/consoleFull)**
for PR 9179 at commit
[`bfd42e1`](https://gith
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150108213
LGTM. Thanks @sun-rui -- Will merge after Jenkins passes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150107792
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150107736
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-150106716
@shivaram, documentation updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does no
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149792487
@shivaram, I will update the documentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149781982
@sun-rui Code change looks pretty good. Could we also make the docs change
in this same PR or were you planning on a separate PR ?
---
If your project is set up for it
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149781578
FWIW Spark config variables are preferred to environment variables if we
are adding something new.
---
If your project is set up for it, you can reply to this email a
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149779214
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149779211
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149779150
**[Test build #44030 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44030/consoleFull)**
for PR 9179 at commit
[`817cc58`](https://git
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149770104
Got it, I don't know what is the plan with env vs. spark conf, but maybe we
want consistency here.
---
If your project is set up for it, you can reply to this email
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149765199
**[Test build #44030 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44030/consoleFull)**
for PR 9179 at commit
[`817cc58`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149764005
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149764074
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149763093
Add safeguard that 'spark.submit.deployMode' may be missing. This is for
consistency with
[SPARK-10711](https://issues.apache.org/jira/browse/SPARK-10711)
---
If your
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149757394
@felixcheung, you are correct, the value of PYSPARK_PYTHON is passed to
workers. In sparkR, "spark.sparkr.r.command" is a spark conf option, and will
be passed to worker
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149755371
Will update the doc until this new conf option is agreed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149755254
There seems some in-consistency for the configurations, some of which are
spark conf options, the other is an env variable. I don't have strong opinion
on this, and try
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149754697
@felixcheung, no, SPARKR_DRIVER_R is for the R shell executable on the
local host, while "spark.sparkr.r.driver.command" is intended for the Rscript
script engine on the
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9179#discussion_r42575713
--- Diff: core/src/main/scala/org/apache/spark/deploy/RRunner.scala ---
@@ -40,7 +40,13 @@ object RRunner {
// Time to wait for SparkR backend t
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149706049
@davies can comment, I believe in PySpark the PYSPARK_DRIVER_PYTHON or
PYSPARK_PYTHON is passed from the driver to the executer
https://github.com/apache/spark/bl
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149698889
Does SPARKR_DRIVER_R map to spark.sparkr.r.driver.command?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/9179#discussion_r42522835
--- Diff: core/src/main/scala/org/apache/spark/deploy/RRunner.scala ---
@@ -40,7 +40,13 @@ object RRunner {
// Time to wait for SparkR backend
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149632684
Yeah this can go under the SparkR section at
http://spark.apache.org/docs/latest/configuration.html#sparkr
---
If your project is set up for it, you can reply to this
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149625288
+1 on a documentation update, otherwise this won't be discoverable. I'll
defer to a reviewer with more R experience for final sign-off and merge.
---
If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149576374
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149576372
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149575992
**[Test build #43974 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43974/consoleFull)**
for PR 9179 at commit
[`3de695c`](https://git
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149562791
thanks for working on this. The changes look fine to me. Obviously need
someone more familiar with the R stuff to review too.
Is there any documentation for
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149533923
**[Test build #43974 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43974/consoleFull)**
for PR 9179 at commit
[`3de695c`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149531788
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9179#issuecomment-149531751
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
GitHub user sun-rui opened a pull request:
https://github.com/apache/spark/pull/9179
[SPARK-10971][SPARKR] RRunner should allow setting path to Rscript.
Add a new spark conf option "spark.sparkr.r.driver.command" to specify the
executable for an R script in client modes.
Th
51 matches
Mail list logo
