[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14744 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/spark/pull/14744#discussion_r76204928 --- Diff: docs/configuration.md --- @@ -1752,6 +1752,14 @@ showDF(properties, numRows = 200, truncate = FALSE) Executable for executing R scripts in client modes for driver. Ignored in cluster modes. + + spark.r.shell.command + R + +Executable for executing sparkR shell in client modes for driver. Ignored in cluster modes. It is the same as environment variable SPARKR_DRIVER_R, but take precedence over it. +spark.r.shell.command is used for interactive mode of sparkR (sparkR shell) while spark.r.driver.command is used for the batch mode (running sparkR script). --- End diff -- Got it :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/14744#discussion_r76203143 --- Diff: docs/configuration.md --- @@ -1752,6 +1752,14 @@ showDF(properties, numRows = 200, truncate = FALSE) Executable for executing R scripts in client modes for driver. Ignored in cluster modes. + + spark.r.shell.command + R + +Executable for executing sparkR shell in client modes for driver. Ignored in cluster modes. It is the same as environment variable SPARKR_DRIVER_R, but take precedence over it. +spark.r.shell.command is used for interactive mode of sparkR (sparkR shell) while spark.r.driver.command is used for the batch mode (running sparkR script). --- End diff -- I think what I mean is `spark.r.shell.command is used for interactive mode of SparkR while...` or `spark.r.shell.command is used for sparkR shell while...` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/spark/pull/14744#discussion_r76174242 --- Diff: docs/configuration.md --- @@ -1752,6 +1752,14 @@ showDF(properties, numRows = 200, truncate = FALSE) Executable for executing R scripts in client modes for driver. Ignored in cluster modes. + + spark.r.shell.command + R + +Executable for executing sparkR shell in client modes for driver. Ignored in cluster modes. It is the same as environment variable SPARKR_DRIVER_R, but take precedence over it. +spark.r.shell.command is used for interactive mode of sparkR (sparkR shell) while spark.r.driver.command is used for the batch mode (running sparkR script). --- End diff -- I mean if I change it to `interactive mode of sparkR shell`, this is a little weird to me as `sparkR shell` already means interactive mode. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/14744#discussion_r76173164 --- Diff: docs/configuration.md --- @@ -1752,6 +1752,14 @@ showDF(properties, numRows = 200, truncate = FALSE) Executable for executing R scripts in client modes for driver. Ignored in cluster modes. + + spark.r.shell.command + R + +Executable for executing sparkR shell in client modes for driver. Ignored in cluster modes. It is the same as environment variable SPARKR_DRIVER_R, but take precedence over it. +spark.r.shell.command is used for interactive mode of sparkR (sparkR shell) while spark.r.driver.command is used for the batch mode (running sparkR script). --- End diff -- Hmm, I'm not entirely following. `sparkR` is the name of the shell script that an user can run from commandline/their shell (lower case) `SparkR` is how we could call R support of Spark (upper case) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/spark/pull/14744#discussion_r76172717 --- Diff: docs/configuration.md --- @@ -1752,6 +1752,14 @@ showDF(properties, numRows = 200, truncate = FALSE) Executable for executing R scripts in client modes for driver. Ignored in cluster modes. + + spark.r.shell.command + R + +Executable for executing sparkR shell in client modes for driver. Ignored in cluster modes. It is the same as environment variable SPARKR_DRIVER_R, but take precedence over it. +spark.r.shell.command is used for interactive mode of sparkR (sparkR shell) while spark.r.driver.command is used for the batch mode (running sparkR script). --- End diff -- here `(sparkR shell)` means `interactive mode of sparkR` and `(running sparkR script)` means `batch mode`. If I use 'sparkR shell' directly, then I might to rephrase the whole senentence. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user zjffdu commented on a diff in the pull request:
https://github.com/apache/spark/pull/14744#discussion_r76172436
--- Diff:
launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
---
@@ -336,7 +336,8 @@
join(File.separator, sparkHome, "R", "lib", "SparkR",
"profile", "shell.R"));
List args = new ArrayList<>();
-args.add(firstNonEmpty(System.getenv("SPARKR_DRIVER_R"), "R"));
+
args.add(firstNonEmpty(firstNonEmpty(conf.get(SparkLauncher.SPARKR_R_SHELL),
--- End diff --
ah, right, my mistake. :(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/14744#discussion_r76171033 --- Diff: docs/configuration.md --- @@ -1752,6 +1752,14 @@ showDF(properties, numRows = 200, truncate = FALSE) Executable for executing R scripts in client modes for driver. Ignored in cluster modes. + + spark.r.shell.command + R + +Executable for executing sparkR shell in client modes for driver. Ignored in cluster modes. It is the same as environment variable SPARKR_DRIVER_R, but take precedence over it. +spark.r.shell.command is used for interactive mode of sparkR (sparkR shell) while spark.r.driver.command is used for the batch mode (running sparkR script). --- End diff -- nit: `sparkR (sparkR shell)` instead, say either "SparkR shell" or "sparkR shell" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14744#discussion_r76170947
--- Diff:
launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
---
@@ -336,7 +336,8 @@
join(File.separator, sparkHome, "R", "lib", "SparkR",
"profile", "shell.R"));
List args = new ArrayList<>();
-args.add(firstNonEmpty(System.getenv("SPARKR_DRIVER_R"), "R"));
+
args.add(firstNonEmpty(firstNonEmpty(conf.get(SparkLauncher.SPARKR_R_SHELL),
--- End diff --
I think you can do this instead:
```
args.add(firstNonEmpty(conf.get(SparkLauncher.SPARKR_R_SHELL),System.getenv("SPARKR_DRIVER_R"),
"R"));
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/14744#discussion_r76097598 --- Diff: docs/configuration.md --- @@ -1752,6 +1752,13 @@ showDF(properties, numRows = 200, truncate = FALSE) Executable for executing R scripts in client modes for driver. Ignored in cluster modes. + + spark.r.shell.command + R + +Executable for executing R shell. --- End diff -- we should clarify how this works with `SPARKR_DRIVER_R` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/14744#discussion_r76083596 --- Diff: docs/configuration.md --- @@ -1752,6 +1752,13 @@ showDF(properties, numRows = 200, truncate = FALSE) Executable for executing R scripts in client modes for driver. Ignored in cluster modes. + + spark.r.shell.command + R + +Executable for executing R shell. --- End diff -- +1 would be good to explain how this is different from `spark.r.shell.command` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14744: [SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set s...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/14744#discussion_r76067417 --- Diff: docs/configuration.md --- @@ -1752,6 +1752,13 @@ showDF(properties, numRows = 200, truncate = FALSE) Executable for executing R scripts in client modes for driver. Ignored in cluster modes. + + spark.r.shell.command + R + +Executable for executing R shell. --- End diff -- maybe "Executable for executing the SparkR shell. Ignored in cluster modes.". Don't have strong opinion on the wording, just be sure to be consistent with the existing description for other 2 related options. The old descriptions can be updated together. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
