[GitHub] spark pull request #22589: [SPARK-25572][SPARKR] test only if not cran
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22589 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22589: [SPARK-25572][SPARKR] test only if not cran
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/22589#discussion_r221440760
--- Diff: R/pkg/tests/run-all.R ---
@@ -18,50 +18,55 @@
library(testthat)
library(SparkR)
-# Turn all warnings into errors
-options("warn" = 2)
+# SPARK-25572
+if (identical(Sys.getenv("NOT_CRAN"), "true")) {
-if (.Platform$OS.type == "windows") {
- Sys.setenv(TZ = "GMT")
-}
+ # Turn all warnings into errors
+ options("warn" = 2)
-# Setup global test environment
-# Install Spark first to set SPARK_HOME
+ if (.Platform$OS.type == "windows") {
+Sys.setenv(TZ = "GMT")
+ }
-# NOTE(shivaram): We set overwrite to handle any old tar.gz files or
directories left behind on
-# CRAN machines. For Jenkins we should already have SPARK_HOME set.
-install.spark(overwrite = TRUE)
+ # Setup global test environment
+ # Install Spark first to set SPARK_HOME
-sparkRDir <- file.path(Sys.getenv("SPARK_HOME"), "R")
-sparkRWhitelistSQLDirs <- c("spark-warehouse", "metastore_db")
-invisible(lapply(sparkRWhitelistSQLDirs,
- function(x) { unlink(file.path(sparkRDir, x), recursive =
TRUE, force = TRUE)}))
-sparkRFilesBefore <- list.files(path = sparkRDir, all.files = TRUE)
+ # NOTE(shivaram): We set overwrite to handle any old tar.gz files or
directories left behind on
+ # CRAN machines. For Jenkins we should already have SPARK_HOME set.
+ install.spark(overwrite = TRUE)
-sparkRTestMaster <- "local[1]"
-sparkRTestConfig <- list()
-if (identical(Sys.getenv("NOT_CRAN"), "true")) {
- sparkRTestMaster <- ""
-} else {
- # Disable hsperfdata on CRAN
- old_java_opt <- Sys.getenv("_JAVA_OPTIONS")
- Sys.setenv("_JAVA_OPTIONS" = paste("-XX:-UsePerfData", old_java_opt))
- tmpDir <- tempdir()
- tmpArg <- paste0("-Djava.io.tmpdir=", tmpDir)
- sparkRTestConfig <- list(spark.driver.extraJavaOptions = tmpArg,
- spark.executor.extraJavaOptions = tmpArg)
-}
+ sparkRDir <- file.path(Sys.getenv("SPARK_HOME"), "R")
+ sparkRWhitelistSQLDirs <- c("spark-warehouse", "metastore_db")
+ invisible(lapply(sparkRWhitelistSQLDirs,
+ function(x) { unlink(file.path(sparkRDir, x), recursive
= TRUE, force = TRUE)}))
+ sparkRFilesBefore <- list.files(path = sparkRDir, all.files = TRUE)
-test_package("SparkR")
+ sparkRTestMaster <- "local[1]"
+ sparkRTestConfig <- list()
+ if (identical(Sys.getenv("NOT_CRAN"), "true")) {
+sparkRTestMaster <- ""
+ } else {
+# Disable hsperfdata on CRAN
+old_java_opt <- Sys.getenv("_JAVA_OPTIONS")
+Sys.setenv("_JAVA_OPTIONS" = paste("-XX:-UsePerfData", old_java_opt))
+tmpDir <- tempdir()
+tmpArg <- paste0("-Djava.io.tmpdir=", tmpDir)
+sparkRTestConfig <- list(spark.driver.extraJavaOptions = tmpArg,
+ spark.executor.extraJavaOptions = tmpArg)
+ }
-if (identical(Sys.getenv("NOT_CRAN"), "true")) {
- # set random seed for predictable results. mostly for base's sample() in
tree and classification
- set.seed(42)
- # for testthat 1.0.2 later, change reporter from "summary" to
default_reporter()
- testthat:::run_tests("SparkR",
- file.path(sparkRDir, "pkg", "tests", "fulltests"),
- NULL,
- "summary")
-}
+ test_package("SparkR")
+
+ if (identical(Sys.getenv("NOT_CRAN"), "true")) {
--- End diff --
We are trying this now - we can clean it up if this works
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22589: [SPARK-25572][SPARKR] test only if not cran
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22589#discussion_r221424365
--- Diff: R/pkg/tests/run-all.R ---
@@ -18,50 +18,55 @@
library(testthat)
library(SparkR)
-# Turn all warnings into errors
-options("warn" = 2)
+# SPARK-25572
+if (identical(Sys.getenv("NOT_CRAN"), "true")) {
-if (.Platform$OS.type == "windows") {
- Sys.setenv(TZ = "GMT")
-}
+ # Turn all warnings into errors
+ options("warn" = 2)
-# Setup global test environment
-# Install Spark first to set SPARK_HOME
+ if (.Platform$OS.type == "windows") {
+Sys.setenv(TZ = "GMT")
+ }
-# NOTE(shivaram): We set overwrite to handle any old tar.gz files or
directories left behind on
-# CRAN machines. For Jenkins we should already have SPARK_HOME set.
-install.spark(overwrite = TRUE)
+ # Setup global test environment
+ # Install Spark first to set SPARK_HOME
-sparkRDir <- file.path(Sys.getenv("SPARK_HOME"), "R")
-sparkRWhitelistSQLDirs <- c("spark-warehouse", "metastore_db")
-invisible(lapply(sparkRWhitelistSQLDirs,
- function(x) { unlink(file.path(sparkRDir, x), recursive =
TRUE, force = TRUE)}))
-sparkRFilesBefore <- list.files(path = sparkRDir, all.files = TRUE)
+ # NOTE(shivaram): We set overwrite to handle any old tar.gz files or
directories left behind on
+ # CRAN machines. For Jenkins we should already have SPARK_HOME set.
+ install.spark(overwrite = TRUE)
-sparkRTestMaster <- "local[1]"
-sparkRTestConfig <- list()
-if (identical(Sys.getenv("NOT_CRAN"), "true")) {
- sparkRTestMaster <- ""
-} else {
- # Disable hsperfdata on CRAN
- old_java_opt <- Sys.getenv("_JAVA_OPTIONS")
- Sys.setenv("_JAVA_OPTIONS" = paste("-XX:-UsePerfData", old_java_opt))
- tmpDir <- tempdir()
- tmpArg <- paste0("-Djava.io.tmpdir=", tmpDir)
- sparkRTestConfig <- list(spark.driver.extraJavaOptions = tmpArg,
- spark.executor.extraJavaOptions = tmpArg)
-}
+ sparkRDir <- file.path(Sys.getenv("SPARK_HOME"), "R")
+ sparkRWhitelistSQLDirs <- c("spark-warehouse", "metastore_db")
+ invisible(lapply(sparkRWhitelistSQLDirs,
+ function(x) { unlink(file.path(sparkRDir, x), recursive
= TRUE, force = TRUE)}))
+ sparkRFilesBefore <- list.files(path = sparkRDir, all.files = TRUE)
-test_package("SparkR")
+ sparkRTestMaster <- "local[1]"
+ sparkRTestConfig <- list()
+ if (identical(Sys.getenv("NOT_CRAN"), "true")) {
+sparkRTestMaster <- ""
+ } else {
+# Disable hsperfdata on CRAN
+old_java_opt <- Sys.getenv("_JAVA_OPTIONS")
+Sys.setenv("_JAVA_OPTIONS" = paste("-XX:-UsePerfData", old_java_opt))
+tmpDir <- tempdir()
+tmpArg <- paste0("-Djava.io.tmpdir=", tmpDir)
+sparkRTestConfig <- list(spark.driver.extraJavaOptions = tmpArg,
+ spark.executor.extraJavaOptions = tmpArg)
+ }
-if (identical(Sys.getenv("NOT_CRAN"), "true")) {
- # set random seed for predictable results. mostly for base's sample() in
tree and classification
- set.seed(42)
- # for testthat 1.0.2 later, change reporter from "summary" to
default_reporter()
- testthat:::run_tests("SparkR",
- file.path(sparkRDir, "pkg", "tests", "fulltests"),
- NULL,
- "summary")
-}
+ test_package("SparkR")
+
+ if (identical(Sys.getenv("NOT_CRAN"), "true")) {
--- End diff --
Eh, should we remove those same conditions?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22589: [SPARK-25572][SPARKR] test only if not cran
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/22589 [SPARK-25572][SPARKR] test only if not cran ## What changes were proposed in this pull request? CRAN doesn't seem to respect the system requirements as running tests - we have seen cases where SparkR is run on Java 10, which unfortunately Spark does not start on. For 2.4, lets attempt skipping all tests ## How was this patch tested? manual, jenkins, appveyor You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark ralltests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22589.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22589 commit 3b7414d9adf55ef74ce2d81403e570e5d6951a05 Author: Felix Cheung Date: 2018-09-29T05:10:23Z test only if not cran --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
