[GitHub] spark issue #22813: [SPARK-25818][CORE] WorkDirCleanup should only remove th...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22813 @ouyangxiaochen . Sorry, but the use case sounds like a misconfiguration. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22830: [SPARK-25838][ML] Remove formatVersion from Savea...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22830#discussion_r228800501 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala --- @@ -163,8 +163,6 @@ class LogisticRegressionModel

[GitHub] spark issue #22830: [SPARK-25838][ML] Remove formatVersion from Saveable

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22830 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22851: [SPARK-25797][SQL][DOCS][BACKPORT-2.3] Add migration doc...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22851 Thank you, @seancxmao and @felixcheung . @seancxmao . Please close this PR since it's merged now

[GitHub] spark issue #22851: [SPARK-25797][SQL][DOCS][BACKPORT-2.3] Add migration doc...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22851 Merged to `branch-2.3`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22861: [SPARK-25663][SPARK-25661][SQL][TEST] Refactor BuiltInDa...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22861 At least, the following is worth for a separate PR because it's orthogonal `Refactor ... to use main method`. ``` 1. Make mainArgs correctly set in BenchmarkBase. ``` One

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22514 I see. Thank you for confirmation, @gatorsmile and @cloud-fan . --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22514 Also, cc @gatorsmile . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22817: [SPARK-25816][SQL] Fix attribute resolution in nested ex...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22817 Thank you, @gatorsmile ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22514 This PR will resolve a regression which is throwing `java.lang.RuntimeException` for Parquet tables. I'm wondering if we can consider this for 2.4.0 RC5

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r228772854 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -2648,7 +2648,7 @@ class DataFrameFunctionsSuite extends

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r228772799 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala --- @@ -21,7 +21,6 @@ import java.net.URI

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r228772719 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSuite.scala --- @@ -92,4 +92,18 @@ class HiveParquetSuite extends QueryTest

[GitHub] spark pull request #22856: [SPARK-25856][SQL][MINOR] Remove AverageLike and ...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22856#discussion_r228771390 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Average.scala --- @@ -23,9 +23,21 @@ import

[GitHub] spark pull request #22788: [SPARK-25769][SQL]escape nested columns by backti...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22788#discussion_r228770171 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -2856,6 +2856,21 @@ class SQLQuerySuite extends QueryTest

[GitHub] spark pull request #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendO...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22617#discussion_r228768273 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArrayBenchmark.scala --- @@ -68,9 +100,7 @@ object

[GitHub] spark pull request #22857: [SPARK-25860][SQL] Replace Literal(null, _) with ...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22857#discussion_r228760320 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -83,6 +83,7 @@ abstract class Optimizer

[GitHub] spark pull request #22857: [SPARK-25860][SQL] Replace Literal(null, _) with ...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22857#discussion_r228760200 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -736,3 +736,65 @@ object CombineConcats

[GitHub] spark pull request #22857: [SPARK-25860][SQL] Replace Literal(null, _) with ...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22857#discussion_r228760023 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseSuite.scala --- @@ -0,0 +1,324

[GitHub] spark pull request #22847: [SPARK-25850][SQL] Make the split threshold for t...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22847#discussion_r228755710 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -812,6 +812,17 @@ object SQLConf { .intConf

[GitHub] spark pull request #22865: [DOC] Fix doc for spark.sql.parquet.recordLevelFi...

2018-10-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22865#discussion_r228755473 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -462,7 +462,7 @@ object SQLConf { val

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2018-10-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19601 Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22843: [SPARK-16693][SPARKR] Remove methods deprecated

2018-10-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22843#discussion_r228616400 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -3477,39 +3447,6 @@ test_that("Window functions on a DataFrame", { expect_eq

[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Improve BenchmarkWideTab...

2018-10-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22823#discussion_r228605621 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BenchmarkWideTable.scala --- @@ -1,52 +0,0 @@ -/* - * Licensed

[GitHub] spark issue #22844: [SPARK-25847][SQL][TEST] Refactor JSONBenchmarks to use ...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22844 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22845: [SPARK-25848][SQL][TEST] Refactor CSVBenchmarks to use m...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22845 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22840: [SPARK-25840][BUILD] `make-distribution.sh` should not f...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22840 Oops. My bad. I'll monitor the branch. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Improve BenchmarkWideTab...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22823#discussion_r228400706 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BenchmarkWideTable.scala --- @@ -1,52 +0,0 @@ -/* - * Licensed

[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Improve BenchmarkWideTab...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22823#discussion_r228399871 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -910,12 +910,14 @@ class

[GitHub] spark pull request #22837: [MINOR][TEST][BRANCH-2.4] Regenerate golden file ...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/22837 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22837: [MINOR][TEST][BRANCH-2.4] Regenerate golden file `dateti...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22837 Merged to `branch-2.4`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22837: [MINOR][TEST][BRANCH-2.4] Regenerate golden file `dateti...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22837 Thank you for review and approval, @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22837: [MINOR][TEST][BRANCH-2.4] Regenerate golden file `dateti...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22837 Could you review this, @HyukjinKwon ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22840: [SPARK-25840][BUILD] `make-distribution.sh` should not f...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22840 Thank you, @srowen and @HyukjinKwon . Merged to master/branch-2.4. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22840: [SPARK-25840][BUILD] `make-distribution.sh` should not f...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22840 @srowen . It's a documented feature. - http://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution I know that you're not against it, but Spark 2.4.0

[GitHub] spark issue #22840: [SPARK-25840][BUILD] `make-distribution.sh` should not f...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22840 I noted that this doesn't aim to reproduce the provided binary release. This aims to help individual contributor participate to test the source artifacts during voting phase

[GitHub] spark pull request #22839: [SPARK-25656][SQL][DOC][EXAMPLE][BRANCH-2.4] Add ...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/22839 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22839: [SPARK-25656][SQL][DOC][EXAMPLE][BRANCH-2.4] Add a doc a...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22839 Thank you so much for review, @dbtsai . Merged to `branch-2.4`. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22840: [SPARK-25840][BUILD] `make-distribution.sh` should not f...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22840 cc @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22840: [SPARK-25840][BUILD] `make-distribution.sh` shoul...

2018-10-25 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/22840 [SPARK-25840][BUILD] `make-distribution.sh` should not fail due to missing LICENSE-binary ## What changes were proposed in this pull request? We vote for the artifacts. All releases

[GitHub] spark pull request #22838: [SPARK-25835][K8s] fix issues with k8s tests

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22838#discussion_r228313490 --- Diff: pom.xml --- @@ -2654,6 +2654,16 @@ kubernetes + +resource-managers/kubernetes/core

[GitHub] spark pull request #22839: [SPARK-25656][SQL][DOC][EXAMPLE][BRANCH-2.4] Add ...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22839#discussion_r228307486 --- Diff: docs/sql-data-sources-load-save-functions.md --- @@ -82,6 +82,49 @@ To load a CSV file you can use: +The extra options

[GitHub] spark pull request #22839: [SPARK-25656][SQL][DOC][EXAMPLE][BRANCH-2.4] Add ...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22839#discussion_r228307138 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -114,10 +114,14 @@ write.df(namesAndAges, "namesAndAges.parquet",

[GitHub] spark pull request #22839: [SPARK-25656][SQL][DOC][EXAMPLE][BRANCH-2.4] Add ...

2018-10-25 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/22839 [SPARK-25656][SQL][DOC][EXAMPLE][BRANCH-2.4] Add a doc and examples about extra data source options ## What changes were proposed in this pull request? Our current doc does

[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTa...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22823#discussion_r228295120 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -910,12 +910,14 @@ class

[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22827 @gatorsmile . Could you put `[BRANCH-2.4]` into PR title? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22837: [MINOR][TEST] Regenerate golden file `datetime.sq...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22837#discussion_r228293689 --- Diff: sql/core/src/test/resources/sql-tests/results/datetime.sql.out --- @@ -82,9 +82,10 @@ struct 1 2 2 3 + -- !query

[GitHub] spark pull request #22837: [MINOR][TEST] Fix datetime.sql.out

2018-10-25 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/22837 [MINOR][TEST] Fix datetime.sql.out ## What changes were proposed in this pull request? `datetime.sql.out` is a generated golden file, but it's a little bit broken during manual

[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22827#discussion_r228269628 --- Diff: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql --- @@ -60,8 +60,3 @@ select zip_with(array('a', 'b', 'c'), array

[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22827#discussion_r228267271 --- Diff: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql --- @@ -60,8 +60,3 @@ select zip_with(array('a', 'b', 'c'), array

[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22827 Also, please remove `public static final int WORD_SIZE = 8;` in UnsafeRow.java. It's added by `map_entries`. - https://github.com/apache/spark/pull/22828/files?utf8=%E2%9C%93=unified=1

[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22827#discussion_r228263430 --- Diff: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql --- @@ -65,21 +65,3 @@ create or replace temporary view nested

[GitHub] spark issue #22828: [SPARK-25832][SQL] Remove newly added map related functi...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22828 Oh, I'm closing this in favor of https://github.com/apache/spark/pull/22827. --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22828: [SPARK-25832][SQL] Remove newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/22828 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22828: [SPARK-25832][SQL] remove newly added map related functi...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22828 To run the Jenkins faster, I create a standalone PR instead of making a PR to #22821 . --- - To unsubscribe, e-mail

[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22828#discussion_r228259220 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java --- @@ -61,8 +61,6 @@ */ public final class

[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22828#discussion_r228258092 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -433,13 +432,9 @@ object FunctionRegistry

[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22828#discussion_r228257902 --- Diff: python/pyspark/sql/functions.py --- @@ -2540,26 +2540,6 @@ def map_values(col): return Column(sc._jvm.functions.map_values

[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22828#discussion_r228257685 --- Diff: R/pkg/NAMESPACE --- @@ -313,7 +313,6 @@ exportMethods("%<=>%", "lower"

[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...

2018-10-25 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/22828 [SPARK-25832][SQL] remove newly added map related functions ## What changes were proposed in this pull request? This aims to supercede https://github.com/apache/spark/pull/22821

[GitHub] spark pull request #22821: [SPARK-25832][SQL] remove newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22821#discussion_r228230625 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -414,7 +414,6 @@ object FunctionRegistry

[GitHub] spark pull request #22821: [SPARK-25832][SQL] remove newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22821#discussion_r228230052 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -414,7 +414,6 @@ object FunctionRegistry

[GitHub] spark pull request #22821: [SPARK-25832][SQL] remove newly added map related...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22821#discussion_r228227982 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -414,7 +414,6 @@ object FunctionRegistry

[GitHub] spark issue #22821: [SPARK-25832][SQL] remove newly added map related functi...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22821 @cloud-fan . I'll update my PR to you once more. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22821: [SPARK-25832][SQL] remove newly added map related functi...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22821 Thank you, @rxin . In that case, +1 for complete removal. It's easier for us to add the expressions back instead of updating exising expressions

[GitHub] spark issue #22821: [SPARK-25832][SQL] remove newly added map related functi...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22821 I'm just confused here. Shall we finish the discussion on the email thread? @cloud-fan and @gatorsmile . If the decision is officially made like that (providing tech. preview to advance users

[GitHub] spark issue #22821: [SPARK-25832][SQL] remove newly added map related functi...

2018-10-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22821 @cloud-fan . That's sounds like a Tech. Preview for the advance users, doesn't it? It looks like an excuse to ignore the whole context of the discussion and to try to ship in any way

[GitHub] spark issue #22819: [BUILD] Close stale PRs

2018-10-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22819 Could you add https://github.com/apache/spark/pull/20405, too? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22811: Branch 2.4

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22811 Hi, @un-knower . Could you close this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21859: [SPARK-24900][SQL]Speed up sort when the dataset ...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/21859#discussion_r227592853 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/SmallDataSortBenchmark.scala --- @@ -0,0 +1,105

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19601 Hi, @kiszk . Can we close this for now? You can make another PR later if you want. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #17174: [SPARK-19145][SQL] Timestamp to String casting is slowin...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17174 @maropu . +1 for your idea. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #16478: [SPARK-7768][SQL] Revise user defined types (UDT)

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16478 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22663: [SPARK-25490][SQL][TEST] Fix OOM of KryoBenchmark due to...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22663 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22804: [SPARK-25665][SQL][TEST] Refactor ObjectHashAggre...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22804#discussion_r227582826 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/execution/benchmark/ObjectHashAggregateExecBenchmark.scala --- @@ -21,207 +21,212 @@ import

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22771 `PagedTableSuite.pageNavigation` failure is fixed in the master. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22771 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22804: [SPARK-25665][SQL][TEST] Refactor ObjectHashAggregateExe...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22804 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22788: [SPARK-25769][SQL]escape nested columns by backtick each...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22788 @viirya . Please see the all four examples. I guess you missed the context. - https://github.com/apache/spark/pull/22788#pullrequestreview-166797320 BTW, I'm good for any methods

[GitHub] spark issue #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and examples ...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22801 Thank you for review, @felixcheung , @dbtsai , @HyukjinKwon . Merged to master. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #22808: [SPARK-25812][UI][TEST] Fix test failure in PagedTableSu...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22808 Thank you, @gengliangwang and @felixcheung . Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22791: [SPARK-25795][R][EXAMPLE] Fix CSV SparkR SQL Exam...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22791#discussion_r227493674 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -114,7 +114,7 @@ write.df(namesAndAges, "namesAndAges.parquet",

[GitHub] spark pull request #22791: [SPARK-25795][R][EXAMPLE] Fix CSV SparkR SQL Exam...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22791#discussion_r227493317 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -114,7 +114,7 @@ write.df(namesAndAges, "namesAndAges.parquet",

[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227478986 --- Diff: examples/src/main/python/sql/datasource.py --- @@ -57,6 +57,15 @@ def basic_datasource_example(spark): format

[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227478845 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -118,6 +118,10 @@ df <- read.df("examples/src/main/resources/people.csv"

[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227457282 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -118,6 +118,10 @@ df <- read.df("examples/src/main/resources/people.csv"

[GitHub] spark pull request #22801: [SPARK-25656][SQL][DOC][EXAMPLE] Add a doc and ex...

2018-10-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227434947 --- Diff: examples/src/main/scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala --- @@ -56,6 +56,13 @@ object SQLDataSourceExample

[GitHub] spark pull request #22801: [SPARK-25656][DOC][EXAMPLE] Add a doc and example...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22801#discussion_r227226704 --- Diff: docs/sql-data-sources-load-save-functions.md --- @@ -82,6 +82,50 @@ To load a CSV file you can use: +The extra options

[GitHub] spark pull request #22801: [SPARK-25656][DOC][EXAMPLE] Add a doc and example...

2018-10-22 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/22801 [SPARK-25656][DOC][EXAMPLE] Add a doc and examples about extra data source options ## What changes were proposed in this pull request? Our current doc does not explain how we

[GitHub] spark issue #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for classNameV2_...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22790 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22788: [SPARK-25769][SQL]escape nested columns by backtick each...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22788 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22791: [SPARK-25795][R][EXAMPLE] Fix CSV SparkR SQL Example

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22791 Thank you for review and merging, @srowen . Merged to `master/branch-2.4/branch-2.3`. --- - To unsubscribe, e-mail

[GitHub] spark issue #22796: [SPARK-25800][Build] Remove all instances of netty 3.x i...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22796 Removing the depedency is good, but currently - It's not clear the effective benefit for Apache Spark by downgrading the dependency in the distribution. - Downgrading in binary

[GitHub] spark pull request #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for class...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22790#discussion_r227152309 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala --- @@ -126,7 +126,7 @@ object BisectingKMeansModel

[GitHub] spark pull request #22788: [SPARK-25769][SQL]change nested columns from `a.b...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22788#discussion_r227093538 --- Diff: sql/core/src/test/resources/sql-tests/results/columnresolution-negative.sql.out --- @@ -81,7 +81,7 @@ SELECT t1.i1 FROM t1, mydb1.t1

[GitHub] spark issue #22796: [SPARK-25800][Build] Remove all instances of netty 3.x i...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22796 Hi, @lipzhu . Could you close this PR and SPARK-25800 issue? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22796: [SPARK-25800][Build] Remove all instances of nett...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22796#discussion_r227043052 --- Diff: dev/deps/spark-deps-hadoop-2.7 --- @@ -148,7 +148,7 @@ metrics-graphite-3.1.5.jar metrics-json-3.1.5.jar metrics-jvm-3.1.5.jar

[GitHub] spark pull request #22791: [SPARK-25795][R][EXAMPLE] Fix CSV SparkR SQL Exam...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22791#discussion_r227041945 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -114,7 +114,7 @@ write.df(namesAndAges, "namesAndAges.parquet",

[GitHub] spark pull request #22791: [SPARK-25795][R][EXAMPLE] Fix CSV SparkR SQL Exam...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22791#discussion_r227014553 --- Diff: examples/src/main/r/RSparkSQLExample.R --- @@ -114,7 +114,7 @@ write.df(namesAndAges, "namesAndAges.parquet",

[GitHub] spark pull request #22792: [SPARK-25795][R][EXAMPLE] Fix CSV SparkR SQL Exam...

2018-10-22 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/22792 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

<    1   2   3   4   5   6   7   8   9   10   >