[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9192 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221952267 Merging this to master and branch-2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221774354 **[Test build #59341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59341/consoleFull)** for PR 9192 at commit [`f67095e`](https://github.com/apache/spark/commit/f67095ef72540140aa2348b5262ffdf91685846a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221774407 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221774409 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59341/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221772896 Thanks for the update. LGTM. Will merge after Jenkins passes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221772591 **[Test build #59341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59341/consoleFull)** for PR 9192 at commit [`f67095e`](https://github.com/apache/spark/commit/f67095ef72540140aa2348b5262ffdf91685846a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221770384 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59337/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221770383 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221770380 **[Test build #59337 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59337/consoleFull)** for PR 9192 at commit [`90641a7`](https://github.com/apache/spark/commit/90641a71ff1860ddfe1a8e0bcb64cc0f0d2a56c6). * This patch **fails R style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221770106 **[Test build #59337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59337/consoleFull)** for PR 9192 at commit [`90641a7`](https://github.com/apache/spark/commit/90641a71ff1860ddfe1a8e0bcb64cc0f0d2a56c6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221768776 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221768777 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59335/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221768774 **[Test build #59335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59335/consoleFull)** for PR 9192 at commit [`640ffca`](https://github.com/apache/spark/commit/640ffcaa2836d40337cda6c3ba1e51f30d16a44c). * This patch **fails R style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221768460 **[Test build #59335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59335/consoleFull)** for PR 9192 at commit [`640ffca`](https://github.com/apache/spark/commit/640ffcaa2836d40337cda6c3ba1e51f30d16a44c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64687334
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -169,48 +169,50 @@ test_that("create DataFrame from RDD", {
error = function(err) {
skip("Hive is not build with SparkSQL, skipped")
})
- sql(hiveCtx, "CREATE TABLE people (name string, age double, height
float)")
- df <- read.df(hiveCtx, jsonPathNa, "json", schema)
+ assign(".sparkRHivesc", hiveCtx, envir = .sparkREnv)
--- End diff --
I'll refactor this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767863 @shivaram it's true there's some scaffolding we need to add (though I'm pretty we could codegen them on the fly instead). I think the idea is this is temporary and in the next release (2.1.0?) we could remove these very easily (just a few lines before and after the method, plus renaming x.default back to x) to stop supporting deprecated calls. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767615 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767617 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59331/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767613 **[Test build #59331 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59331/consoleFull)** for PR 9192 at commit [`98e7ab9`](https://github.com/apache/spark/commit/98e7ab978dd1a7aba06ee47eb012c496f00cbe3b). * This patch **fails R style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221767353 **[Test build #59331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59331/consoleFull)** for PR 9192 at commit [`98e7ab9`](https://github.com/apache/spark/commit/98e7ab978dd1a7aba06ee47eb012c496f00cbe3b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64686844
--- Diff: R/pkg/R/SQLContext.R ---
@@ -362,12 +433,14 @@ sql <- function(sqlContext, sqlQuery) {
#' sc <- sparkR.init()
#' sqlContext <- sparkRSQL.init(sc)
#' path <- "path/to/file.json"
-#' df <- read.json(sqlContext, path)
+#' df <- read.json(path)
#' registerTempTable(df, "table")
-#' new_df <- tableToDF(sqlContext, "table")
+#' new_df <- tableToDF("table")
#' }
+#' @note since 2.0.0
--- End diff --
not yet, I don;t want to keep checking it, that would be the next PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221766378 LGTM except some minor comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64686315
--- Diff: R/pkg/R/SQLContext.R ---
@@ -254,6 +301,7 @@ jsonFile <- function(sqlContext, path) {
#' df <- jsonRDD(sqlContext, rdd)
#'}
+# TODO: remove - this method is no longer exported
--- End diff --
opened SPARK-15545
one reason to not remove this right away is because we have been talking
about calling internal methods via `SparkR:::jsonRDD` for this and other RDD
methods.
we have .Deprecated() here though for a warning
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64686229
--- Diff: R/pkg/R/jobj.R ---
@@ -77,6 +77,11 @@ print.jobj <- function(x, ...) {
cat("Java ref type", name, "id", x$id, "\n", sep = " ")
}
+getClassName.jobj <- function(x) {
--- End diff --
Could you address this comment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64686152
--- Diff: R/pkg/R/SQLContext.R ---
@@ -37,6 +37,37 @@ getInternalType <- function(x) {
stop(paste("Unsupported type for SparkDataFrame:", class(x
}
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
--- End diff --
I think this is very specific to this file - this helper is implemented to
specifically check & remove sqlContext parameter; I'll add more documentation
on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64685560
--- Diff: R/pkg/R/SQLContext.R ---
@@ -362,12 +433,14 @@ sql <- function(sqlContext, sqlQuery) {
#' sc <- sparkR.init()
#' sqlContext <- sparkRSQL.init(sc)
#' path <- "path/to/file.json"
-#' df <- read.json(sqlContext, path)
+#' df <- read.json(path)
#' registerTempTable(df, "table")
-#' new_df <- tableToDF(sqlContext, "table")
+#' new_df <- tableToDF("table")
#' }
+#' @note since 2.0.0
--- End diff --
remove?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64685428
--- Diff: R/pkg/R/SQLContext.R ---
@@ -362,12 +433,14 @@ sql <- function(sqlContext, sqlQuery) {
#' sc <- sparkR.init()
#' sqlContext <- sparkRSQL.init(sc)
#' path <- "path/to/file.json"
-#' df <- read.json(sqlContext, path)
+#' df <- read.json(path)
#' registerTempTable(df, "table")
-#' new_df <- tableToDF(sqlContext, "table")
+#' new_df <- tableToDF("table")
#' }
+#' @note since 2.0.0
--- End diff --
?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64675714
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -169,48 +169,50 @@ test_that("create DataFrame from RDD", {
error = function(err) {
skip("Hive is not build with SparkSQL, skipped")
})
- sql(hiveCtx, "CREATE TABLE people (name string, age double, height
float)")
- df <- read.df(hiveCtx, jsonPathNa, "json", schema)
+ assign(".sparkRHivesc", hiveCtx, envir = .sparkREnv)
--- End diff --
I think this might go away if we are moving to SparkSession
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221676892 Thanks @felixcheung for the update. I left some minor comments inline. It seems unfortunate that we need to do some amount of code duplication to get this to work (i.e. define `read.df` and `read.df.default` etc.) But I think thats fine for two reasons (a) this is an internal code issue and we can continue to clean it up (b) i dont think we are adding a lot of methods there -- in fact we should remove some of the unused ones. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64633002
--- Diff: R/pkg/R/SQLContext.R ---
@@ -37,6 +37,37 @@ getInternalType <- function(x) {
stop(paste("Unsupported type for SparkDataFrame:", class(x
}
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
--- End diff --
can we move this to utils.R. Also some function level comments on what the
arguments mean would be useful (for example `numFuncSig` is only used to print
the deprecation warning from what i see)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64633530
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -169,48 +169,50 @@ test_that("create DataFrame from RDD", {
error = function(err) {
skip("Hive is not build with SparkSQL, skipped")
})
- sql(hiveCtx, "CREATE TABLE people (name string, age double, height
float)")
- df <- read.df(hiveCtx, jsonPathNa, "json", schema)
+ assign(".sparkRHivesc", hiveCtx, envir = .sparkREnv)
--- End diff --
minor nit: we should add a new method to create the test hive context that
also does this assignment. seems like something that other test cases might
forget to do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r64633209
--- Diff: R/pkg/R/SQLContext.R ---
@@ -254,6 +301,7 @@ jsonFile <- function(sqlContext, path) {
#' df <- jsonRDD(sqlContext, rdd)
#'}
+# TODO: remove - this method is no longer exported
--- End diff --
Can we open a JIRA for this ? Would be good to clean up this file as I
think a bunch of functions are not exported here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221130256 Took me a while to rebase, and caught up with new changes. This is WIP - I will need to check on roxygen2 doc, and look into SparkSession --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221130044 **[Test build #59164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59164/consoleFull)** for PR 9192 at commit [`4884e56`](https://github.com/apache/spark/commit/4884e56efe19233ce35574d2aeeebb7d3213e20e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221130085 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221130088 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59164/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221130082 **[Test build #59164 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59164/consoleFull)** for PR 9192 at commit [`4884e56`](https://github.com/apache/spark/commit/4884e56efe19233ce35574d2aeeebb7d3213e20e). * This patch **fails some tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-22418 SparkSession definitely makes more sense, given DataFrame is the main API ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221103064 Sorry I was busy last week and missed this -- but +1 to keeping backwards compatibility. BTW on that note will this also change the entry point in SparkR to be SparkSession (instead of SQL/SparkContext that is) ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-220816684 I think maintaining backward compatibility would be useful. Do you have time to bring this up to date this week so we get it in for 2.0? It's pretty late for a change of this size, but given this is a huge usability improvement and R is experimental, it might be ok. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-220767086 Would you like API without SQLContext (or SparkSession) parameter or, as what is in this PR, API that can be backward compatible with or without it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-220490142 Actually I'd love to have this...! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-152774654 Any more thought on this and #9185 ? It looks like we are cutting the 1.6.0 release very soon, and it will be good if API changes can go into a minor release change --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42772732
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
+ # Strip sqlContext from list of parameters and then pass the rest along.
+ # In the following, if '&' is used instead of '&&', it warns about
+ # "the condition has length > 1 and only the first element will be used"
+ if (class(x) == "jobj" &&
+ grepl("org.apache.spark.sql.SQLContext", capture.output(show(x {
+.Deprecated(newFuncSig, old = paste0(funcName, "(sqlContext...)"))
+f(...)
+ } else {
+f(x, ...)
+ }
+}
--- End diff --
Yeah this is related to the discussion in
https://github.com/apache/spark/pull/9185 as well I guess. Lets discuss this
point in #9185 and based on that we can update this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-150165422 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-150165423 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44139/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-150165301 **[Test build #44139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44139/consoleFull)** for PR 9192 at commit [`2f9e50d`](https://github.com/apache/spark/commit/2f9e50d8530b8f2010480b4f534c84863874973f). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42726488
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
+ # Strip sqlContext from list of parameters and then pass the rest along.
+ # In the following, if '&' is used instead of '&&', it warns about
+ # "the condition has length > 1 and only the first element will be used"
+ if (class(x) == "jobj" &&
+ grepl("org.apache.spark.sql.SQLContext", capture.output(show(x {
+.Deprecated(newFuncSig, old = paste0(funcName, "(sqlContext...)"))
+f(...)
+ } else {
+f(x, ...)
+ }
+}
--- End diff --
I took a rough look at https://github.com/apache/spark/pull/8909, it seems
that it is possible to have multiple root SQLContexts if
"spark.sql.allowMultipleContexts" is true. Even there is only one root
SQLContext (when "spark.sql.allowMultipleContexts" is false), there could be
multiple session SQLContexts (created by call rootSQLContext.newSession or
rootHiveContext.newSession()).
I am very clear out session management of SQLContext, @davies, could you
give me your point here? I am thinking do we need to expose session support in
SparkR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42726208
--- Diff: R/pkg/R/jobj.R ---
@@ -77,6 +77,11 @@ print.jobj <- function(x, ...) {
cat("Java ref type", name, "id", x$id, "\n", sep = " ")
}
+getClassName.jobj <- function(x) {
--- End diff --
print.jobj can be updated to use this method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-150128523 **[Test build #44139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44139/consoleFull)** for PR 9192 at commit [`2f9e50d`](https://github.com/apache/spark/commit/2f9e50d8530b8f2010480b4f534c84863874973f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42716848
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
--- End diff --
changed. thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-150127311 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-150127349 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42711959
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
+ # Strip sqlContext from list of parameters and then pass the rest along.
+ # In the following, if '&' is used instead of '&&', it warns about
+ # "the condition has length > 1 and only the first element will be used"
+ if (class(x) == "jobj" &&
+ grepl("org.apache.spark.sql.SQLContext", capture.output(show(x {
+.Deprecated(newFuncSig, old = paste0(funcName, "(sqlContext...)"))
+f(...)
+ } else {
+f(x, ...)
+ }
+}
--- End diff --
OK. I am neutral on this point. @shivaram, what's your opinion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42711903
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
--- End diff --
I am using R 3.1.1, and find no get0 in the base package. It seems that
get0 is added in a later R version? If so, either we change our supported R
version, or we don't use get0 for version compatibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149995526 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44080/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149995521 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149995360 **[Test build #44080 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44080/consoleFull)** for PR 9192 at commit [`2c16ca8`](https://github.com/apache/spark/commit/2c16ca839e3721118d2534cc7f5e9fa7ac58297f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149990149 **[Test build #44080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44080/consoleFull)** for PR 9192 at commit [`2c16ca8`](https://github.com/apache/spark/commit/2c16ca839e3721118d2534cc7f5e9fa7ac58297f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149989430 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149989384 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42664614
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
+ # Strip sqlContext from list of parameters and then pass the rest along.
+ # In the following, if '&' is used instead of '&&', it warns about
+ # "the condition has length > 1 and only the first element will be used"
+ if (class(x) == "jobj" &&
+ grepl("org.apache.spark.sql.SQLContext", capture.output(show(x {
--- End diff --
updated. thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42662379
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
+ # Strip sqlContext from list of parameters and then pass the rest along.
+ # In the following, if '&' is used instead of '&&', it warns about
+ # "the condition has length > 1 and only the first element will be used"
+ if (class(x) == "jobj" &&
+ grepl("org.apache.spark.sql.SQLContext", capture.output(show(x {
+.Deprecated(newFuncSig, old = paste0(funcName, "(sqlContext...)"))
+f(...)
+ } else {
+f(x, ...)
+ }
+}
--- End diff --
The proposal for this is to eliminate the sqlContext parameter from
SQLContext-parity methods in R. Primarily this makes methods friendlier in R
and more R-like (eg. read.df()). The changed method signature would be the one
we would like to keep in the next release.
Reasons for this have been discussed in JIRA, but to recap:
1. We only support one sqlContext in R - and having multiple at a time can
be very confusing (eg. table not accessible)
2. For hiveCtx vs sqlContext, hiveCtx is preferred
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42659746
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
--- End diff --
get0 is in {base} right?
https://stat.ethz.ch/R-manual/R-devel/library/base/html/exists.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/9192#discussion_r42659643 --- Diff: R/pkg/R/SQLContext.R --- @@ -17,6 +17,34 @@ # SQLcontext.R: SQLContext-driven functions +#' Temporary function to reroute old S3 Method call to new --- End diff -- "reroute" was the term corresponding to "dispatch" "temporary" was referring to the fact that we intend this to go away - please see my other answer regarding your question on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42602160
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
+ # Strip sqlContext from list of parameters and then pass the rest along.
+ # In the following, if '&' is used instead of '&&', it warns about
+ # "the condition has length > 1 and only the first element will be used"
+ if (class(x) == "jobj" &&
+ grepl("org.apache.spark.sql.SQLContext", capture.output(show(x {
+.Deprecated(newFuncSig, old = paste0(funcName, "(sqlContext...)"))
+f(...)
+ } else {
+f(x, ...)
+ }
+}
--- End diff --
Thought: dispatchFunc strips sqlContext and calls into .default methods. Is
it better for dispatchFunc to add sqlContext if it does not exist and calls
into .default methods which accepts sqlContext as its first argument? This
allows a use case that user can pass in different sqlContext, say a sqlContext
and a hiveContext. It seems that sqlContext and hiveContext can co-exist?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42601678
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
+ # Strip sqlContext from list of parameters and then pass the rest along.
+ # In the following, if '&' is used instead of '&&', it warns about
+ # "the condition has length > 1 and only the first element will be used"
+ if (class(x) == "jobj" &&
+ grepl("org.apache.spark.sql.SQLContext", capture.output(show(x {
--- End diff --
Could we add a getClassName method to the jobj class and use it here
instead of grepl?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9192#discussion_r42601604
--- Diff: R/pkg/R/SQLContext.R ---
@@ -17,6 +17,34 @@
# SQLcontext.R: SQLContext-driven functions
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
+ funcName <- as.character(sys.call(sys.parent())[[1]])
+ f <- get0(paste0(funcName, ".default"))
--- End diff --
seems get0 is not in base package. Could we use get so that we won't
introduce dependency on an extension package?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/9192#discussion_r42601519 --- Diff: R/pkg/R/SQLContext.R --- @@ -17,6 +17,34 @@ # SQLcontext.R: SQLContext-driven functions +#' Temporary function to reroute old S3 Method call to new --- End diff -- wording: Temporary -> Dispatching ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149806172 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44042/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149806170 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149806069 [Test build #44042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44042/console) for PR 9192 at commit [`d8e91f3`](https://github.com/apache/spark/commit/d8e91f375ee7cc33ba77d67634aebbe201e9391f). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149793990 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44047/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149793988 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149793919 **[Test build #44047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44047/consoleFull)** for PR 9192 at commit [`b0348d7`](https://github.com/apache/spark/commit/b0348d71a421b48aa435e11d7e134e66f92e7aec). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149791993 **[Test build #44047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44047/consoleFull)** for PR 9192 at commit [`b0348d7`](https://github.com/apache/spark/commit/b0348d71a421b48aa435e11d7e134e66f92e7aec). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149791268 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149791254 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149791103 Rebase to master. Update to fix the new `as.DataFrame` method. @davies Thanks! Appreciated. I'd like to leave R doc update separated if that's ok by you - too many files to change and too many possible conflicts. It would be easier as a doc only PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149790122 @felixcheung This looks great overall, could you also update all the examples? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149782827 [Test build #44042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44042/consoleFull) for PR 9192 at commit [`d8e91f3`](https://github.com/apache/spark/commit/d8e91f375ee7cc33ba77d67634aebbe201e9391f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149782082 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149782098 cc @davies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-149782068 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/9192 [SPARK-10903] [SPARKR] R - Simplify SQLContext method signatures and use a singleton Eliminate the need to pass sqlContext to method since it is a singleton - and we don't want to support multiple contexts in a R session. Changes are done in a back compat way with deprecation warning added. Method signature for S3 methods are added in a concise, clean approach such that in the next release the deprecated signature can be taken out easily/cleanly (just delete a few lines per method). Custom method dispatch is implemented to allow for multiple JVM reference types that are all 'jobj' in R and to avoid having to add 30 new exports. You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rsqlcontext Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9192.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9192 commit 612f7f3613f881e164f76d0b937fb4787736d2ee Author: felixcheung Date: 2015-10-09T21:31:14Z Refractor SQLContext and DataFrame functions to lookup sqlContext instance in the env commit efedce53a315d7ce23a53145e3de100d2a471690 Author: felixcheung Date: 2015-10-20T06:31:44Z Method dispatch to support omission of 'sqlContext' argument commit fa50f789208e6a00c17b99c2deee8bbfbf091f4e Author: felixcheung Date: 2015-10-20T22:31:07Z Improve route logic commit fd3a835c2eea03d57b3ab6e5ae68f6497569e513 Author: felixcheung Date: 2015-10-20T22:35:20Z update tests commit 8b3141a023e836e31be50bbb2d7f199711440015 Author: felixcheung Date: 2015-10-21T01:01:38Z Change to method dispatch update more tests and add tests for back compat commit d8e91f375ee7cc33ba77d67634aebbe201e9391f Author: felixcheung Date: 2015-10-21T01:17:12Z fix test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
