[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221786544 @shivaram I will create a JIRA soon. Thursday and Friday, I will be on travel to NYC. Will do it on Saturday. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13284 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221772558 Yeah thats a good idea @wangmiao1981 can you open a JIRA to not mask `startsWith` and `endsWith` by updating our generics ? LGTM - Merging this to master and branch-2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221772080 looks fine - I think we should really try to make startsWith and endsWith work though, but that could be a follow up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221668874 @shivaram I am debugging and try to find a hint. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221663730 @wangmiao1981 Lets continue the pipeRDD debugging on the JIRA. This change LGTM for the subset and the masking tests @felixcheung any other comments ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221656579 Re-tested on Ubuntu, the pipedRDD test case still fails. R version 3.3.0 beta (2016-03-30 r70404) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221641136 I comment out the failed case. There is no other failures. As mentioned by @shivaram , we can try to fix the pipeRDD in a separate JIRA. I suspect it is related to operating system. It passed when I use ubuntu, but always fails if I use my mac. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221505799 I did try to solve the PipeRDD issue, but just as @wangmiao1981 said, "For the failure pipedRDD test case, if I copy & paste it in the SparkR, it works fine." It's so weird. I guess it is related to recent R changes, possibly 3.2.4 or else. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221502917 now I'm seeing the pipeRDD test failure... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221489454 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221489457 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59263/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221489377 **[Test build #59263 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59263/consoleFull)** for PR 13284 at commit [`3028e2c`](https://github.com/apache/spark/commit/3028e2c419e55e4cfa2c743361ca8b0beb004de3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221487340 @vectorijk please run conflicts(detail = TRUE) and check the $package:SparkR with test_context.R, namesOfMasked. 3.2.0 should have more methods than 3.1.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221486881 **[Test build #59263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59263/consoleFull)** for PR 13284 at commit [`3028e2c`](https://github.com/apache/spark/commit/3028e2c419e55e4cfa2c743361ca8b0beb004de3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user vectorijk commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221486634 > @shivaram I will make the change with R version check. @wangmiao1981 FYI, I have switched to R 3.1.3 before on Mac. It seems fail too. Do you mind try it again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221486545 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59260/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221486544 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221486456 **[Test build #59260 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59260/consoleFull)** for PR 13284 at commit [`4a187ca`](https://github.com/apache/spark/commit/4a187cacc04bb43b9399698fa95eb4e322b706e7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221486087 @vectorijk As we discussed above, R version is different on local and Jenkins. We installed R 3.3.0 in local while Jenkins still uses the old version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user vectorijk commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221485421 We may also investigate why unit test run differently on Jenkins and local. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13284#discussion_r64520919
--- Diff: R/pkg/inst/tests/testthat/test_context.R ---
@@ -24,10 +24,18 @@ test_that("Check masked functions", {
func <- lapply(masked, function(x) { capture.output(showMethods(x))[[1]]
})
funcSparkROrEmpty <- grepl("\\(package SparkR\\)$|^$", func)
maskedBySparkR <- masked[funcSparkROrEmpty]
- namesOfMasked <- c("describe", "cov", "filter", "lag", "na.omit",
"predict", "sd", "var",
- "colnames", "colnames<-", "intersect", "rank",
"rbind", "sample", "subset",
- "summary", "transform", "drop", "window",
"as.data.frame",
- "endsWith", "startsWith")
+ if (as.numeric(R.version$major) == 3 && as.numeric(R.version$minor) > 2)
{
--- End diff --
Sounds good. Let me revise it. Just started learning R.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221483871 Ok - lets just fix the masking and the subset ones in this PR. Once we understand what is the problem with `pipeRDD` we can fix it in a separate PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/13284#discussion_r64520757
--- Diff: R/pkg/inst/tests/testthat/test_context.R ---
@@ -24,10 +24,18 @@ test_that("Check masked functions", {
func <- lapply(masked, function(x) { capture.output(showMethods(x))[[1]]
})
funcSparkROrEmpty <- grepl("\\(package SparkR\\)$|^$", func)
maskedBySparkR <- masked[funcSparkROrEmpty]
- namesOfMasked <- c("describe", "cov", "filter", "lag", "na.omit",
"predict", "sd", "var",
- "colnames", "colnames<-", "intersect", "rank",
"rbind", "sample", "subset",
- "summary", "transform", "drop", "window",
"as.data.frame",
- "endsWith", "startsWith")
+ if (as.numeric(R.version$major) == 3 && as.numeric(R.version$minor) > 2)
{
--- End diff --
can we just append to the list ? i.e. create `namesOfMasked` outside this
and if the version number is greater than 2 we can do
`namesOfMasked <- c("endsWith", "startsWith", namesOfMasked)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221483796 **[Test build #59260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59260/consoleFull)** for PR 13284 at commit [`4a187ca`](https://github.com/apache/spark/commit/4a187cacc04bb43b9399698fa95eb4e322b706e7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221483353 @shivaram @felixcheung For the failure pipedRDD test case, if I copy & paste it in the SparkR, it works fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221478629 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221478607 **[Test build #59256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59256/consoleFull)** for PR 13284 at commit [`5ac9b9b`](https://github.com/apache/spark/commit/5ac9b9ba3094714f1d153a3f96690fa312b794ae). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221478631 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59256/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221477942 as for subset - maybe tighter signature matching checks in R 3.3.0? I couldn't find any reference on that though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221477791 so startsWith endsWith were definitely added in 3.3.0. I think we should be able to change the generic to match the signature to get it to work (have not tried it yet, since I don't have 3.3.0) ``` startsWith(x, prefix) endsWith(x, suffix) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221477450 Ok - so that explains one of the problems. Does anybody know what the problem in `subset` is ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221477072 @shivaram I will make the change with R version check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221476748 Continuing from my previous message, we can't add or remove `endsWith` and `startsWith` as we want to support all R versions from 3.1.0 onwards. We could get the R version at run time (using ` R.Version()$major`) and only add them for 3.3.0 and higher --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221476475 **[Test build #59256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59256/consoleFull)** for PR 13284 at commit [`5ac9b9b`](https://github.com/apache/spark/commit/5ac9b9ba3094714f1d153a3f96690fa312b794ae). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221476116 Hmm looks like `startsWith` and `endsWith` were added in R 3.3.0 - See http://www.r-statistics.com/2016/05/r-3-3-0-is-released/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221476011 @shivaram The pipedRDD one seems working when using sudo in Linux. My mac does not work though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221475769 @felixcheung > conflicts(detail = TRUE) $.GlobalEnv [1] "df" $`package:SparkR` [1] "alias" "cov" "filter""fitted" [5] "glm" "lag" "na.omit" "predict" [9] "sd""var" "window""head" [13] "str" "show" "%in%" "as.data.frame" [17] "atan2" "attach""colnames" "colnames<-" [21] "drop" "endsWith" "factorial" "ifelse" [25] "intersect" "mean" "merge" "ncol" [29] "nrow" "rank" "rbind" "sample" [33] "startsWith""subset""substr""summary" [37] "transform" "unique""with" $`package:stats` [1] "alias" "cov" "df" "filter" "fitted" "glm" "lag" [8] "na.omit" "predict" "sd" "var" "window" $`package:utils` [1] "head" "str" $`package:methods` [1] "show" "body<-""kronecker" $`package:base` [1] "%in%" "as.data.frame" "atan2" "attach" [5] "body<-""colnames" "colnames<-""drop" [9] "endsWith" "factorial" "ifelse""intersect" [13] "kronecker" "mean" "merge" "ncol" [17] "nrow" "rank" "rbind" "sample" [21] "startsWith""subset""substr""summary" [25] "transform" "unique""with" See the $`package:SparkR` above --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221475397 R version 3.3.0 (2016-05-03) -- "Supposedly Educational" Copyright (C) 2016 The R Foundation for Statistical Computing Platform: x86_64-apple-darwin13.4.0 (64-bit) > installed.packages() Package SparkR "SparkR" base "base" boot "boot" class "class" cluster"cluster" codetools "codetools" compiler "compiler" crayon "crayon" datasets "datasets" digest "digest" foreign"foreign" graphics "graphics" grDevices "grDevices" grid "grid" KernSmooth "KernSmooth" lattice"lattice" magrittr "magrittr" MASS "MASS" Matrix "Matrix" memoise"memoise" methods"methods" mgcv "mgcv" nlme "nlme" nnet "nnet" parallel "parallel" praise "praise" R6 "R6" rpart "rpart" spatial"spatial" splines"splines" stats "stats" stats4 "stats4" survival "survival" tcltk "tcltk" testthat "testthat" tools "tools" utils "utils" LibPath SparkR "/Users/mwang/spark_ws_0904/R/lib" base "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" boot "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" class "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" cluster"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" codetools "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" compiler "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" crayon "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" datasets "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" digest "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" foreign"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" graphics "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" grDevices "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" grid "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" KernSmooth "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" lattice"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" magrittr "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" MASS "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" Matrix "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" memoise"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" methods"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" mgcv "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" nlme "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" nnet "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" parallel "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" praise "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" R6 "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" rpart "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" spatial"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" splines"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" stats "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" stats4 "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" survival "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" tcltk "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" testthat "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" tools "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" utils "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" Version Priority SparkR "2.0.0" NA base "3.3.0" "base" boot "1.3-18" "recommended" class "7.3-14" "recommended" cluster"2.0.4" "recommended" codetools "0.2-14" "recommended" compiler "3.3.0" "base" crayon "1.3.1" NA datasets "3.3.0" "base" digest "0.6.9" NA foreign"0.8-66" "recommended" graphics "3.3.0" "base" grDevices "3.3.0" "base" grid
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13284#discussion_r64513464
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1445,7 +1445,11 @@ setMethod("[", signature(x = "SparkDataFrame"),
#' }
setMethod("subset", signature(x = "SparkDataFrame"),
function(x, subset, select, drop = F, ...) {
-x[subset, select, drop = drop]
+if (missing(subset)) {
+x[, select, drop = drop]
--- End diff --
I think we should keep the ..., like this
`x[, select, drop = drop, ...]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13284#discussion_r64513468
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1445,7 +1445,11 @@ setMethod("[", signature(x = "SparkDataFrame"),
#' }
setMethod("subset", signature(x = "SparkDataFrame"),
function(x, subset, select, drop = F, ...) {
-x[subset, select, drop = drop]
+if (missing(subset)) {
+x[, select, drop = drop]
+} else {
+x[subset, select, drop = drop]
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13284#discussion_r64513429
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1445,7 +1445,11 @@ setMethod("[", signature(x = "SparkDataFrame"),
#' }
setMethod("subset", signature(x = "SparkDataFrame"),
function(x, subset, select, drop = F, ...) {
-x[subset, select, drop = drop]
+if (missing(subset)) {
+x[, select, drop = drop]
+} else {
--- End diff --
could you pls fix the extra space: `else {`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221469622 I'm seeing the error with `subset` as well. As for "endsWith", "startsWith" - could you check what you have in your environment - you could run `conflicts(detail = TRUE)` - AFAIK I'm not seeing this. Which version of R do you have? And package versions? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221462859 @wangmiao1981 Thanks for investigating this. Do you know why these are not failing in Jenkins though ? (the subset test and the pipedRDD one) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221426358 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221426332 **[Test build #59228 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59228/consoleFull)** for PR 13284 at commit [`b385009`](https://github.com/apache/spark/commit/b3850093a7ae1c3c356afb64aa78b2f069127618). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221426362 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59228/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221422898 **[Test build #59228 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59228/consoleFull)** for PR 13284 at commit [`b385009`](https://github.com/apache/spark/commit/b3850093a7ae1c3c356afb64aa78b2f069127618). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
GitHub user wangmiao1981 opened a pull request:
https://github.com/apache/spark/pull/13284
[SPARK-15439][SparkR]:Failed to run unit test in SparkR
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
There are some failures when running SparkR unit tests.
In this PR, I fixed two of these failures in test_context.R and
test_sparkSQL.R
The first one is due to different masked name. I added missed names in the
expected arrays.
The second one is because one PR removed the logic of a previous fix of
missing subset method.
The file privilege issue is still there. I am debugging it. SparkR shell
can run the test case successfully.
test_that("pipeRDD() on RDDs", {
actual <- collect(pipeRDD(rdd, "more"))
When using run-test script, it complains no such directories as below:
cannot open file '/tmp/Rtmp4FQbah/filee2273f9d47f7': No such file or
directory
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
Manually test it
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangmiao1981/spark R
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13284.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13284
commit b3850093a7ae1c3c356afb64aa78b2f069127618
Author: wm...@hotmail.com
Date: 2016-05-24T22:29:49Z
fix Unit test error
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
