[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113650123 LGTM, waiting for tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113656892 [Test build #35316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35316/console) for PR 6889 at commit [`9ce9f1e`](https://github.com/apache/spark/commit/9ce9f1ea0fd19209fd543a0650a20b46901d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113656998 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6889 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113631102 [Test build #35316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35316/consoleFull) for PR 6889 at commit [`9ce9f1e`](https://github.com/apache/spark/commit/9ce9f1ea0fd19209fd543a0650a20b46901d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113630396 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113630386 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113663815 Thanks @falaki -- Merging this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6889#discussion_r32842867
--- Diff: R/pkg/R/sparkR.R ---
@@ -278,3 +278,32 @@ sparkRHive.init - function(jsc = NULL) {
assign(.sparkRHivesc, hiveCtx, envir = .sparkREnv)
hiveCtx
}
+
+#' Assigns a group ID to all the jobs started by this thread until the
group ID is set to a
+#' different value or cleared.
+#'
+#' @param sc existing spark context
+#' @param groupid the ID to be assigned to job groups
+#' @param description description for the the job group ID
+#' @param interruptOnCancel flag to indicate if the job is interrupted on
job cancellation
--- End diff --
Yeah @davies point is a good one. We can add an example usage here with
something like
```
#' @examples
#'\dontrun{
#' sc - sparkR.init()
# setJobGroup(sc, group, some group, TRUE)
#'}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113338106 [Test build #35193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35193/console) for PR 6889 at commit [`a2c19af`](https://github.com/apache/spark/commit/a2c19af62bf7ea259f121aa4ab9ba9971e806f90). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113338130 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113370880 LGTM, it will be better if you could add some tests for it, and add examples in the doc. Right now it's hard to tell what's the type of these parameters. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6889#discussion_r32786101
--- Diff: R/pkg/R/sparkR.R ---
@@ -278,3 +278,38 @@ sparkRHive.init - function(jsc = NULL) {
assign(.sparkRHivesc, hiveCtx, envir = .sparkREnv)
hiveCtx
}
+
+#' Assigns a group ID to all the jobs started by this thread until the
group ID is set to a
+#' different value or cleared.
+#'
+#' @param sc The existing
+#' @param groupid the ID to be assigned to job groups
+#' @param description description for the the job group ID
+#' @param interruptOnCancel flag to indicate if the job is interrupted on
job cancellation
+
+setJobGroup - function(groupId, description, interruptOnCancel) {
+ if (exists(.sparkRjsc, envir = env)) {
--- End diff --
While its technically fine to just lookup the `SparkContext` I think all
our methods take in a SQLContext / SparkContext explicitly. Will that work for
your use case as well ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/6889#discussion_r32786324
--- Diff: R/pkg/R/sparkR.R ---
@@ -278,3 +278,38 @@ sparkRHive.init - function(jsc = NULL) {
assign(.sparkRHivesc, hiveCtx, envir = .sparkREnv)
hiveCtx
}
+
+#' Assigns a group ID to all the jobs started by this thread until the
group ID is set to a
+#' different value or cleared.
+#'
+#' @param sc The existing
+#' @param groupid the ID to be assigned to job groups
+#' @param description description for the the job group ID
+#' @param interruptOnCancel flag to indicate if the job is interrupted on
job cancellation
+
+setJobGroup - function(groupId, description, interruptOnCancel) {
+ if (exists(.sparkRjsc, envir = env)) {
--- End diff --
Yes, that is perfectly fine. Just wondering why doesn't sparkR.stop()
follow that convention?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
GitHub user falaki opened a pull request: https://github.com/apache/spark/pull/6889 [SPARKR][SPARK-8452] expose jobGroup API in SparkR This pull request adds following methods to SparkR: ```R setJobGroup() cancelJobGroup() clearJobGroup() ``` For each method we recover the existing java spark context. There does not seem to be a good way to test these in R. cc @shivaram and @davies You can merge this pull request into a Git repository by running: $ git pull https://github.com/falaki/spark SPARK-8452 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6889.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6889 commit 343ca771f45fb1d8e8cff255ec143b7182674011 Author: Hossein hoss...@databricks.com Date: 2015-06-18T22:03:04Z Added setJobGroup, cancelJobGroup and clearJobGroup to SparkR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113304260 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113304283 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113304452 [Test build #35181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35181/consoleFull) for PR 6889 at commit [`343ca77`](https://github.com/apache/spark/commit/343ca771f45fb1d8e8cff255ec143b7182674011). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113321776 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113321783 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113322063 [Test build #35193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35193/consoleFull) for PR 6889 at commit [`a2c19af`](https://github.com/apache/spark/commit/a2c19af62bf7ea259f121aa4ab9ba9971e806f90). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113324276 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARKR][SPARK-8452] expose jobGroup API in Sp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6889#issuecomment-113324170 [Test build #35181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35181/console) for PR 6889 at commit [`343ca77`](https://github.com/apache/spark/commit/343ca771f45fb1d8e8cff255ec143b7182674011). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
