[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user olarayej commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147776388 @sun-rui @felixcheung Folks: is there anything else we'd need to do on this one? If not, could we merge this function and close this PR? Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9031 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147782794 LGTM. Thanks @adrian555 for the change and @sun-rui and @felixcheung for the reviews. Merging this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147179242 [Test build #43540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43540/consoleFull) for PR 9031 at commit [`99a5422`](https://github.com/apache/spark/commit/99a54223a3b2456db1412e303bbe3f3b3c90e662). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147179851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43540/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147179850 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147179029 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147178999 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147179020 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41698039 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what +cols <- columns(bf) +newEnv <- new.env() +for (i in 1:length(cols)) { --- End diff -- @shivaram @felixcheung Anyone knows how to create a DataFrame with 0 column? Could you please show me how to do that? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41699843 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what +cols <- columns(bf) +newEnv <- new.env() +for (i in 1:length(cols)) { --- End diff -- I'm not sure we can do that now from R: ``` > a <- list() > b <- createDataFrame(sqlContext, a) Error in take(x, 1)[[1]] : subscript out of bounds > a <- data.frame() > a data frame with 0 columns and 0 rows > b <- createDataFrame(sqlContext, a) Error in FUN(X[[i]], ...) : can not infer type from NULL ``` Or 0-col DataFrame being very useful? Though I agree we could check for it, in case that support is added in the future. I'd suggest either adding a `stopifnot()` or a test case for that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41699442 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- not for `bf` ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147096207 @sun-rui First, the current subsetting function '$' of DataFrame ends up with a 'Column' class instead of a real column access of 'DataFrame'. So to operate on one column of a DataFrame, in SparkR, you will have to use df[, "columnName"] instead. Second, the major reason of using attach() is to access the columns of a DataFrame directly by adding the column name to the search list for R so that R can find it just by its name. We see customers using this in their scripts often. If SparkR has this too, it will make SparkR more R friendly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41701074 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- but it is an internal variable.. :) Ok, I will remove it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147146800 Hi admin, The failure is [error] (core/*:mimaPreviousClassfiles) sbt.ResolveException: download failed: org.apache.spark#spark-unsafe_2.10;1.5.0!spark-unsafe_2.10.jar [error] Total time: 144 s, completed Oct 10, 2015 1:44:17 PM could you please investigate and rerun the test, please? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147122953 [Test build #43532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43532/consoleFull) for PR 9031 at commit [`99a5422`](https://github.com/apache/spark/commit/99a54223a3b2456db1412e303bbe3f3b3c90e662). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147124202 [Test build #43532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43532/console) for PR 9031 at commit [`99a5422`](https://github.com/apache/spark/commit/99a54223a3b2456db1412e303bbe3f3b3c90e662). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147124210 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147124212 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43532/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41701060 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what +cols <- columns(bf) +newEnv <- new.env() +for (i in 1:length(cols)) { --- End diff -- k, I will add a stopifnot. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147122665 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147122669 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41694025 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what +cols <- columns(bf) +newEnv <- new.env() +for (i in 1:length(cols)) { --- End diff -- I think so: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala#L267. But I have not tried it:) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147042514 [Test build #43520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43520/consoleFull) for PR 9031 at commit [`99e712e`](https://github.com/apache/spark/commit/99e712ea343b1f3d88674a4c2316bd6c8479eeae). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147042831 [Test build #43519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43519/console) for PR 9031 at commit [`0563a85`](https://github.com/apache/spark/commit/0563a85d2a31e573765b4fc8e8be9b9e62136828). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147042847 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147045228 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43520/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147045205 [Test build #43520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43520/console) for PR 9031 at commit [`99e712e`](https://github.com/apache/spark/commit/99e712ea343b1f3d88674a4c2316bd6c8479eeae). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147045227 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147046856 @adrian555, yes, attach() is useful in R. What I mean is that supporting attach() for DataFrame by allowing direct use of column name is so useful? User can simply use $? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147042848 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43519/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41692013 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- bf is not necessary. and the naming is confusing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41692037 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what +cols <- columns(bf) +newEnv <- new.env() +for (i in 1:length(cols)) { + assign(x=cols[i], value=bf[,cols[i]], envir=newEnv) --- End diff -- x=, value= are not necessary. style: x[, cols[i]] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41692030 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what +cols <- columns(bf) +newEnv <- new.env() +for (i in 1:length(cols)) { --- End diff -- 1:length(cols) has problem if the data frame has 0 column --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41692008 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { --- End diff -- function(x, --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41691995 --- Diff: R/pkg/inst/tests/test_sparkSQL.R --- @@ -1362,6 +1362,26 @@ test_that("Method as.data.frame as a synonym for collect()", { expect_equal(as.data.frame(irisDF2), collect(irisDF2)) }) +test_that("attach() on a DataFrame", { + df <- jsonFile(sqlContext, jsonPath) + expect_error(age) + attach(df) + expect_is(age, "DataFrame") + expected_age <- data.frame(age=c(NA, 30, 19)) --- End diff -- style: age = c(NA, 30, 19) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41691992 --- Diff: R/pkg/inst/tests/test_sparkSQL.R --- @@ -1362,6 +1362,26 @@ test_that("Method as.data.frame as a synonym for collect()", { expect_equal(as.data.frame(irisDF2), collect(irisDF2)) }) +test_that("attach() on a DataFrame", { + df <- jsonFile(sqlContext, jsonPath) + expect_error(age) + attach(df) + expect_is(age, "DataFrame") + expected_age <- data.frame(age=c(NA, 30, 19)) + expect_equal(head(age), expected_age) + stat <- summary(age) + expect_equal(collect(stat)[5, "age"], "30") + age <- age$age + 1 + expect_is(age, "Column") + rm(age) + stat2 <- summary(age) + expect_equal(collect(stat2)[5, "age"], "30") + detach("df") + stat3 <- summary(df[,"age"]) --- End diff -- style: df[, "age"] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41692004 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), --- End diff -- generally, we use (x = "DataFrame") --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147024333 @adrian555, could you explain more on this feature? What is your use case for accessing a col in a DataFrame by directly using its name and the benefit? attach() accepts a data.frame, so does attach(collect(df), ...) meets your requirement? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147041458 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147041453 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41693561 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- The naming is all required by R. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147042246 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147042241 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147042281 @sun-rui The attach() function is a very useful R function and very commonly used by customers. The link to the R function is https://stat.ethz.ch/R-manual/R-devel/library/base/html/attach.html. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147042040 [Test build #43519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43519/consoleFull) for PR 9031 at commit [`0563a85`](https://github.com/apache/spark/commit/0563a85d2a31e573765b4fc8e8be9b9e62136828). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41693574 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what +cols <- columns(bf) +newEnv <- new.env() +for (i in 1:length(cols)) { --- End diff -- The input class for bf is DataFrame. Can a SparkR DataFrame have 0 column? If not, then it is unnecessary to add extra validation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41693577 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what +cols <- columns(bf) +newEnv <- new.env() +for (i in 1:length(cols)) { + assign(x=cols[i], value=bf[,cols[i]], envir=newEnv) --- End diff -- I will update to follow the style. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41693517 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), --- End diff -- For generic function, you have to follow the R function's parameter name, which is 'what'. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/9031#discussion_r41693523 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,33 @@ setMethod("as.data.frame", } collect(x) }) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { --- End diff -- Please see my answer to previous comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-147028556 @sun-rui this PR was closed/changed from these two and my earlier comments there: https://github.com/apache/spark/pull/8985 https://github.com/apache/spark/pull/8997 And this is the R attach function: https://stat.ethz.ch/R-manual/R-devel/library/base/html/attach.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146943788 reviewed earlier PR. looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146962533 @felixcheung Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146613845 @adrian555 Could you bring this up to date with master branch ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146709745 cc @sun-rui --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146672326 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146682793 Tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146672328 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43421/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146672195 [Test build #43421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43421/console) for PR 9031 at commit [`283b91c`](https://github.com/apache/spark/commit/283b91c9bbe5a1c9142d1d1089d61ed8f16360c0). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class BinaryHashJoinNode(` * `case class BroadcastHashJoinNode(` * `trait HashJoinNode ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-14559 @shivaram I have created the new pull request #9031 with the up-to-date files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
GitHub user adrian555 opened a pull request: https://github.com/apache/spark/pull/9031 [SPARK-10913][SPARKR] attach() function support Bring the change code up to date. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adrian555/spark attach2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9031.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9031 commit 283b91c9bbe5a1c9142d1d1089d61ed8f16360c0 Author: adrian555Date: 2015-10-08T19:36:22Z [SPARK-10913][SPARKR] attach() function support --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146667220 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146667176 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9031#issuecomment-146668226 [Test build #43421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43421/consoleFull) for PR 9031 at commit [`283b91c`](https://github.com/apache/spark/commit/283b91c9bbe5a1c9142d1d1089d61ed8f16360c0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146669787 Ok thanks. We shouldn't need to open a new PR for this and you can just push a merge to this same branch and this PR should get updated. Anyways, could you close this PR to avoid confusion ? Only the PR author has permission to do this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146671198 Thanks @shivaram. First time doing so. :) I will follow your suggestion next time. Closing this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 closed the pull request at: https://github.com/apache/spark/pull/8997 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146269857 Hi, anything else needed from me to have this PR delivered? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/8985#issuecomment-145951987 if we add attach shouldn't we add detach? https://stat.ethz.ch/R-manual/R-devel/library/base/html/attach.html Also, isn't `with` better for this so as to not leak the reference --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-145962605 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43286/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-145962536 [Test build #43286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43286/console) for PR 8997 at commit [`804052f`](https://github.com/apache/spark/commit/804052f416f49929ad9e97fef98ec27d68b6800b). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/8997#discussion_r41307264 --- Diff: R/pkg/inst/tests/test_sparkSQL.R --- @@ -1341,6 +1341,15 @@ test_that("Method as.data.frame as a synonym for collect()", { expect_equal(as.data.frame(irisDF2), collect(irisDF2)) }) +test_that("attach() on a DataFrame", { + df <- jsonFile(sqlContext, jsonPath) + expect_error(age) + attach(df) + expect_output(age, "DataFrame[age:bigint]", fixed=T) --- End diff -- from the other PR: more test please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/8997#discussion_r41307679 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,34 @@ setMethod("as.data.frame", collect(x) } ) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- should this be `df`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/8985#discussion_r41301106 --- Diff: R/pkg/inst/tests/test_sparkSQL.R --- @@ -1327,6 +1327,15 @@ test_that("SQL error message is returned from JVM", { expect_equal(grepl("Table Not Found: blah", retError), TRUE) }) +test_that("attach() on a DataFrame", { + df <- jsonFile(sqlContext, jsonPath) + expect_error(age) + attach(df) + expect_output(age, "DataFrame[age:bigint]", fixed=T) --- End diff -- could you please add more test on how it could be used? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
GitHub user adrian555 opened a pull request: https://github.com/apache/spark/pull/8997 [SPARK-10913][SPARKR] attach() function support You can merge this pull request into a Git repository by running: $ git pull https://github.com/adrian555/spark attach1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8997.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8997 commit 804052f416f49929ad9e97fef98ec27d68b6800b Author: adrian555Date: 2015-10-06T18:10:37Z [SPARK-10913][SPARKR] attach() function support --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/8985#issuecomment-145955245 I created a new pull request #8997 to address Shivaram's comment. @felixcheung the R native detach() works so we do not need SparkR version of detach(). The with() is another function I am planning to work on after this one. I think we need both. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-145956720 [Test build #43286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43286/consoleFull) for PR 8997 at commit [`804052f`](https://github.com/apache/spark/commit/804052f416f49929ad9e97fef98ec27d68b6800b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 closed the pull request at: https://github.com/apache/spark/pull/8985 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8985#issuecomment-145978403 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43280/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8985#issuecomment-145978252 [Test build #43280 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43280/console) for PR 8985 at commit [`0aa94a2`](https://github.com/apache/spark/commit/0aa94a2f4b1fbab975d6061db111e01fb9ccce2c). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/8997#discussion_r41319707 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,34 @@ setMethod("as.data.frame", collect(x) } ) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- Is there a mandatory requirement to use df instead of bf as internal variable in the code? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8985#issuecomment-145978401 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-145953575 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-145953513 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-145962603 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/8997#discussion_r41338088 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,34 @@ setMethod("as.data.frame", collect(x) } ) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/8997#discussion_r4105 --- Diff: R/pkg/inst/tests/test_sparkSQL.R --- @@ -1341,6 +1341,15 @@ test_that("Method as.data.frame as a synonym for collect()", { expect_equal(as.data.frame(irisDF2), collect(irisDF2)) }) +test_that("attach() on a DataFrame", { + df <- jsonFile(sqlContext, jsonPath) + expect_error(age) + attach(df) + expect_output(age, "DataFrame[age:bigint]", fixed=T) --- End diff -- I have added more tests to test_sparkSQL.R. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146027147 Hi admin, I have added some more test cases, could you please rerun? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146027649 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146027677 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146029314 [Test build #43304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43304/consoleFull) for PR 8997 at commit [`5f3f262`](https://github.com/apache/spark/commit/5f3f2623ba996a420406ee3d7ca824cb58d7b0b0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146031812 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146031725 [Test build #43304 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43304/console) for PR 8997 at commit [`5f3f262`](https://github.com/apache/spark/commit/5f3f2623ba996a420406ee3d7ca824cb58d7b0b0). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8997#issuecomment-146031813 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43304/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/8997#discussion_r41336332 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,34 @@ setMethod("as.data.frame", collect(x) } ) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- just not sure why `bf`? `df` might be more consistent or readable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/8997#discussion_r41336776 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,34 @@ setMethod("as.data.frame", collect(x) } ) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- Oh, bf originally stands for 'big frame' :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user adrian555 commented on a diff in the pull request: https://github.com/apache/spark/pull/8997#discussion_r41336967 --- Diff: R/pkg/R/DataFrame.R --- @@ -1881,3 +1881,34 @@ setMethod("as.data.frame", collect(x) } ) + +#' The specified DataFrame is attached to the R search path. This means that +#' the DataFrame is searched by R when evaluating a variable, so columns in +#' the DataFrame can be accessed by simply giving their names. +#' +#' @rdname attach +#' @title Attach DataFrame to R search path +#' @param what (DataFrame) The DataFrame to attach +#' @param pos (integer) Specify position in search() where to attach. +#' @param name (character) Name to use for the attached DataFrame. Names +#' starting with package: are reserved for library. +#' @param warn.conflicts (logical) If TRUE, warnings are printed about conflicts +#' from attaching the database, unless that DataFrame contains an object +#' @examples +#' \dontrun{ +#' attach(irisDf) +#' summary(Sepal_Width) +#' } +#' @seealso \link{detach} +setMethod("attach", + signature(what = "DataFrame"), + function(what, pos = 2, name = deparse(substitute(what)), warn.conflicts = TRUE) { +bf <- what --- End diff -- And sometime I try to avoid 'df' since df is a valid R stats function (the F Distribution). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/8985#issuecomment-145931583 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8985#issuecomment-145932076 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10913][SPARKR] attach() function suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8985#issuecomment-145932103 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org