[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20681 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20681#discussion_r171910649 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -67,6 +67,8 @@ sparkSession <- if (windows_with_hadoop()) { sparkR.session(master = sparkRTestMaster, enableHiveSupport = FALSE) } sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession) +# materialize the catalog implementation +listTables() --- End diff -- ok --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...
Github user liufengdb commented on a diff in the pull request: https://github.com/apache/spark/pull/20681#discussion_r171770982 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -67,6 +67,8 @@ sparkSession <- if (windows_with_hadoop()) { sparkR.session(master = sparkRTestMaster, enableHiveSupport = FALSE) } sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession) +# materialize the catalog implementation +listTables() --- End diff -- `test_sparkSQL.R` is the only one uses `newJObject("org.apache.spark.sql.hive.test.TestHiveContext", ssc, FALSE)` on the `ssc`, so the catalog impl spark conf is changed. So ``test_sparkSQL.R` is the only one broken. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20681#discussion_r171750940 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -67,6 +67,8 @@ sparkSession <- if (windows_with_hadoop()) { sparkR.session(master = sparkRTestMaster, enableHiveSupport = FALSE) } sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession) +# materialize the catalog implementation +listTables() --- End diff -- we are calling `sparkR.session(master = sparkRTestMaster, enableHiveSupport = FALSE)` is almost every other test files - does the same apply in those other places? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20681#discussion_r170862425 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -885,24 +894,24 @@ test_that("collect() and take() on a DataFrame return the same number of rows an }) test_that("collect() support Unicode characters", { - lines <- c("{\"name\":\"ìë íì¸ì\"}", - "{\"name\":\"æ¨å¥½\", \"age\":30}", - "{\"name\":\"ããã«ã¡ã¯\", \"age\":19}", - "{\"name\":\"Xin chà o\"}") + lines <- c("{\"name\":\"?\"}", + "{\"name\":\"??\", \"age\":30}", + "{\"name\":\"?\", \"age\":19}", + "{\"name\":\"Xin ch?o\"}") --- End diff -- no, we should not change these --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/20681#discussion_r170840835 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -885,24 +894,24 @@ test_that("collect() and take() on a DataFrame return the same number of rows an }) test_that("collect() support Unicode characters", { - lines <- c("{\"name\":\"ìë íì¸ì\"}", - "{\"name\":\"æ¨å¥½\", \"age\":30}", - "{\"name\":\"ããã«ã¡ã¯\", \"age\":19}", - "{\"name\":\"Xin chà o\"}") + lines <- c("{\"name\":\"?\"}", + "{\"name\":\"??\", \"age\":30}", + "{\"name\":\"?\", \"age\":19}", + "{\"name\":\"Xin ch?o\"}") --- End diff -- Is it okay to replace these unicode characters? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20681#discussion_r170787801 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -46,16 +49,19 @@ setHiveContext <- function(sc) { }) hiveSession <- callJMethod(hiveCtx, "sparkSession") } - previousSession <- get(".sparkRsession", envir = .sparkREnv) assign(".sparkRsession", hiveSession, envir = .sparkREnv) assign(".prevSparkRsession", previousSession, envir = .sparkREnv) + assign(".prevSessionConf", previousConf, envir = .sparkREnv) hiveSession } unsetHiveContext <- function() { previousSession <- get(".prevSparkRsession", envir = .sparkREnv) + previousConf <- get(".prevSessionConf", envir = .sparkREnv) + callJStatic("org.apache.spark.sql.api.r.SQLUtils", "setSparkContextSessionConf", previousSessioni, previousConf) --- End diff -- this should be `previousSession`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20681: [SPARK-23518][SQL] Avoid metastore access when th...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20681#discussion_r170787695 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -46,16 +49,19 @@ setHiveContext <- function(sc) { }) hiveSession <- callJMethod(hiveCtx, "sparkSession") } - previousSession <- get(".sparkRsession", envir = .sparkREnv) assign(".sparkRsession", hiveSession, envir = .sparkREnv) assign(".prevSparkRsession", previousSession, envir = .sparkREnv) + assign(".prevSessionConf", previousConf, envir = .sparkREnv) hiveSession } unsetHiveContext <- function() { previousSession <- get(".prevSparkRsession", envir = .sparkREnv) + previousConf <- get(".prevSessionConf", envir = .sparkREnv) + callJStatic("org.apache.spark.sql.api.r.SQLUtils", "setSparkContextSessionConf", previousSessioni, previousConf) assign(".sparkRsession", previousSession, envir = .sparkREnv) remove(".prevSparkRsession", envir = .sparkREnv) + remove(".prevSparkConf", envir = .sparkREnv) --- End diff -- this should be `. prevSessionConf`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org