[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r227675871 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -434,17 +426,34 @@ object ScalaReflection extends

[GitHub] spark issue #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder to get ...

2018-10-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder to get ...

2018-10-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 Let me rebase again. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r227655503 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala --- @@ -45,6 +46,11 @@ case class

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r227640284 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -635,13 +675,17 @@ object ScalaReflection extends

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r227638126 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -128,6 +128,16 @@ object ScalaReflection extends

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r227638782 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -622,6 +654,14 @@ object ScalaReflection extends

[GitHub] spark issue #22807: [WIP][SPARK-25811][PySpark] Raise a proper error when un...

2018-10-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22807 @BryanCutler Thanks for looking at this! Yea, this is a WIP work for early review and I will add a config to toggle the `safe` flag later

[GitHub] spark pull request #22807: [WIP][SPARK-25811][PySpark] Raise a proper error ...

2018-10-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22807#discussion_r227611310 --- Diff: python/pyspark/serializers.py --- @@ -248,7 +248,14 @@ def create_array(s, t): # TODO: see ARROW-2432. Remove when the minimum

[GitHub] spark pull request #22807: [WIP][SPARK-25811][PySpark] Raise a proper error ...

2018-10-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22807#discussion_r227611338 --- Diff: python/pyspark/sql/tests.py --- @@ -4961,6 +4961,31 @@ def foofoo(x, y): ).collect ) +def

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-10-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22514 @cloud-fan The high level idea is not to put expose conversion details to `CreateTable`. But let `CreateHiveTableAsSelectCommand` to decide whether to do conversion. So in `HiveAnalysis` rule

[GitHub] spark issue #22788: [SPARK-25769][SQL]escape nested columns by backtick each...

2018-10-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22788 @dongjoon-hyun Oh I see. The ambiguousness is in the results of `sql` for several inputs. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #22807: [WIP][SPARK-25811][PySpark] Raise a proper error when un...

2018-10-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22807 cc @HyukjinKwon @BryanCutler --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22807: [WIP][SPARK-25811][PySpark] Raise a proper error ...

2018-10-23 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22807 [WIP][SPARK-25811][PySpark] Raise a proper error when unsafe cast is detected by PyArrow ## What changes were proposed in this pull request? Since 0.11.0, PyArrow supports to raise

[GitHub] spark issue #22788: [SPARK-25769][SQL]escape nested columns by backtick each...

2018-10-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22788 From above examples, ```scala scala> $"`a.b`".expr.asInstanceOf[org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute].sql res1: String = `a.b`

[GitHub] spark pull request #22754: [SPARK-25776][CORE][MINOR]The disk write buffer s...

2018-10-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22754#discussion_r227315262 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java --- @@ -62,6 +62,8 @@ public UnsafeSorterSpillWriter

[GitHub] spark issue #22754: [SPARK-25776][CORE][MINOR]The disk write buffer size mus...

2018-10-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22754 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22754: [SPARK-25776][CORE][MINOR]The disk write buffer size mus...

2018-10-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22754 You can remove [MINOR] from the title since there is a JIRA ticket now. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22785: [SPARK-25791][SQL] Datatype of serializers in Row...

2018-10-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22785#discussion_r227314368 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala --- @@ -187,7 +187,7 @@ object RowEncoder { val

[GitHub] spark pull request #22785: [SPARK-25791][SQL] Datatype of serializers in Row...

2018-10-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22785#discussion_r227310290 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/RowEncoderSuite.scala --- @@ -273,6 +273,16 @@ class RowEncoderSuite extends

[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...

2018-10-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22787 Thanks @HyukjinKwon @dongjoon-hyun --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for class...

2018-10-22 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22790#discussion_r227210362 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala --- @@ -126,7 +126,7 @@ object BisectingKMeansModel extends

[GitHub] spark pull request #22754: [SPARK-25776][CORE][MINOR]The disk write buffer s...

2018-10-22 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22754#discussion_r227198037 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java --- @@ -62,6 +62,8 @@ public UnsafeSorterSpillWriter

[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22787 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for class...

2018-10-22 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22790#discussion_r227188654 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala --- @@ -126,7 +126,7 @@ object BisectingKMeansModel extends

[GitHub] spark issue #22798: [SPARK-25803] Fix docker-image-tool.sh -n option

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22798 cc @vanzin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder to get ...

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 hmm, seems even re-open still can't re-trigger AppVeyor... cc @HyukjinKwon do you know why? --- - To unsubscribe, e-mail

[GitHub] spark issue #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder to get ...

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 Close and reopen this to re-trigger AppVeyor. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-22 Thread viirya
GitHub user viirya reopened a pull request: https://github.com/apache/spark/pull/22749 [SPARK-25746][SQL] Refactoring ExpressionEncoder to get rid of flat flag ## What changes were proposed in this pull request? This is inspired during implementing #21732. For now

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-22 Thread viirya
Github user viirya closed the pull request at: https://github.com/apache/spark/pull/22749 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder to get ...

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22787 Seems github is restored... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22787 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22662: [SPARK-25627][TEST] Reduce test time for ContinuousStres...

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22662 Thanks @srowen @dongjoon-hyun @kiszk --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22787 Thanks @HyukjinKwon. I've fixed that and committed. But looks like Github is out of order now. The commit is now disappeared. Let's wait for Github backing to normal. :) On Mon

[GitHub] spark issue #22655: [SPARK-25666][PYTHON] Internally document type conversio...

2018-10-22 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22655 @HyukjinKwon Cool! Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for class...

2018-10-22 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22790#discussion_r226980682 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala --- @@ -126,7 +126,7 @@ object BisectingKMeansModel extends

[GitHub] spark pull request #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for class...

2018-10-22 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22790#discussion_r226980495 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala --- @@ -126,7 +126,7 @@ object BisectingKMeansModel extends

[GitHub] spark pull request #22785: [SPARK-25791][SQL] Datatype of serializers in Row...

2018-10-22 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22785#discussion_r226979620 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/RowEncoderSuite.scala --- @@ -273,6 +273,16 @@ class RowEncoderSuite extends

[GitHub] spark issue #22788: [SPARK-25769][SQL]change nested columns from `a.b` to `a...

2018-10-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22788 @huaxingao Can you update the PR description based on @dongjoon-hyun's comments https://github.com/apache/spark/pull/22788#pullrequestreview-166797320. It is more clear on the improvement

[GitHub] spark issue #22788: [SPARK-25769][SQL]change nested columns from `a.b` to `a...

2018-10-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22788 @huaxingao Can you update the PR description based on @dongjoon-hyun's comments https://github.com/apache/spark/pull/22788#pullrequestreview-166797320. It is more clear on the improvement

[GitHub] spark issue #22788: [SPARK-25769][SQL]change nested columns from `a.b` to `a...

2018-10-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22788 @huaxingao Can you update the PR description based on @dongjoon-hyun's comments https://github.com/apache/spark/pull/22788#pullrequestreview-166797320. It is more clear on the improvement

[GitHub] spark pull request #22787: [SPARK-25040][SQL] Empty string for non string ty...

2018-10-21 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22787#discussion_r226884649 --- Diff: docs/sql-migration-guide-upgrade.md --- @@ -11,6 +11,8 @@ displayTitle: Spark SQL Upgrading Guide - In PySpark, when creating

[GitHub] spark pull request #22787: [SPARK-25040][SQL] Empty string for non string ty...

2018-10-21 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22787#discussion_r226884659 --- Diff: docs/sql-migration-guide-upgrade.md --- @@ -11,6 +11,8 @@ displayTitle: Spark SQL Upgrading Guide - In PySpark, when creating

[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...

2018-10-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22787 Yea, I do think so. I will update it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22785: [SPARK-25791][SQL] Datatype of serializers in RowEncoder...

2018-10-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22785 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...

2018-10-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22787 cc @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22787: [SPARK-25040][SQL] Empty string for non string ty...

2018-10-21 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22787 [SPARK-25040][SQL] Empty string for non string types should be disallowed ## What changes were proposed in this pull request? This takes over original PR at #22019. The original proposal

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r226846925 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -43,10 +44,11 @@ import

[GitHub] spark pull request #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEnc...

2018-10-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r226844557 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala --- @@ -198,7 +196,7 @@ object RowEncoder

[GitHub] spark pull request #22785: [SPARK-25791][SQL] Datatype of serializers in Row...

2018-10-20 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22785 [SPARK-25791][SQL] Datatype of serializers in RowEncoder should be accessible ## What changes were proposed in this pull request? The serializers of `RowEncoder` use few `If` Catalyst

[GitHub] spark issue #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEncoder to...

2018-10-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 cc @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEncoder to...

2018-10-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 hmm, I don't touch PySpark files. Why the building fails at PySpark pip packaging tests... --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEncoder to...

2018-10-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEnc...

2018-10-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r226828791 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala --- @@ -198,7 +196,7 @@ object RowEncoder

[GitHub] spark issue #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEncoder to...

2018-10-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22662: [SPARK-25627][TEST] Reduce test time for ContinuousStres...

2018-10-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22662 ping @tdas and @zsxwing Can you take a look this? Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #22764: [SPARK-25765][ML] Add training cost to BisectingK...

2018-10-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22764#discussion_r226807614 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala --- @@ -225,13 +227,14 @@ object BisectingKMeansModel extends

[GitHub] spark pull request #22764: [SPARK-25765][ML] Add training cost to BisectingK...

2018-10-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22764#discussion_r226688846 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala --- @@ -225,13 +227,14 @@ object BisectingKMeansModel extends

[GitHub] spark issue #22773: [MINOR][SQL] Add prettyNames for from_json, to_json, fro...

2018-10-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22773 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22773: [MINOR][SQL] Add prettyNames for from_json, to_json, fro...

2018-10-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22773 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22764: [SPARK-25765][ML] Add training cost to BisectingK...

2018-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22764#discussion_r226516595 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala --- @@ -225,13 +227,14 @@ object BisectingKMeansModel extends

[GitHub] spark pull request #22764: [SPARK-25765][ML] Add training cost to BisectingK...

2018-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22764#discussion_r226516200 --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala --- @@ -310,4 +317,6 @@ class BisectingKMeansSummary private[clustering

[GitHub] spark pull request #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEnc...

2018-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r226506566 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -43,10 +44,11 @@ import

[GitHub] spark pull request #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEnc...

2018-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r226506718 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -103,75 +88,61 @@ object ExpressionEncoder

[GitHub] spark pull request #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEnc...

2018-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r226507104 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -212,21 +183,88 @@ object ExpressionEncoder

[GitHub] spark pull request #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEnc...

2018-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r226506593 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -103,75 +88,61 @@ object ExpressionEncoder

[GitHub] spark pull request #22764: [SPARK-25765][ML] Add training cost to BisectingK...

2018-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22764#discussion_r226358124 --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala --- @@ -310,4 +317,6 @@ class BisectingKMeansSummary private[clustering

[GitHub] spark pull request #22708: [SPARK-21402][SQL] Fix java array of structs dese...

2018-10-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225834919 --- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithMapSuite.java --- @@ -0,0 +1,257 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #22708: [SPARK-21402][SQL] Fix java array of structs dese...

2018-10-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225820031 --- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java --- @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEncoder to...

2018-10-17 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22708: [SPARK-21402][SQL] Fix java array of structs dese...

2018-10-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225769471 --- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java --- @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #22708: [SPARK-21402][SQL] Fix java array of structs dese...

2018-10-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225768857 --- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java --- @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #22745: [SPARK-21402][SQL][FOLLOW-UP] Fix java map of str...

2018-10-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22745#discussion_r225768707 --- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithMapSuite.java --- @@ -0,0 +1,257 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #22707: [SPARK-25717][SQL] Insert overwrite a recreated e...

2018-10-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22707#discussion_r225759293 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -227,18 +227,22 @@ case class InsertIntoHiveTable

[GitHub] spark issue #22708: [SPARK-21402][SQL] Fix java array of structs deserializa...

2018-10-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22708 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22749: [WIP][SPARK-25746][SQL] Refactoring ExpressionEnc...

2018-10-16 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22749 [WIP][SPARK-25746][SQL] Refactoring ExpressionEncoder to get rid of flat flag ## What changes were proposed in this pull request? This is inspired during implementing #21732. For now

[GitHub] spark issue #22708: [SPARK-21402] Fix java array of structs deserialization

2018-10-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22708 And please add [SQL] to the PR title. Like `[SPARK-21402][SQL]` --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22708: [SPARK-21402] Fix java array of structs deseriali...

2018-10-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225454136 --- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java --- @@ -0,0 +1,195 @@ +package test.org.apache.spark.sql

[GitHub] spark pull request #22708: [SPARK-21402] Fix java array of structs deseriali...

2018-10-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225453267 --- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java --- @@ -0,0 +1,195 @@ +package test.org.apache.spark.sql

[GitHub] spark pull request #22708: [SPARK-21402] Fix java array of structs deseriali...

2018-10-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225452879 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala --- @@ -26,6 +26,7 @@ import scala.language.existentials

[GitHub] spark issue #22708: [SPARK-21402] Fix java array/map of structs deserializat...

2018-10-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22708 Please modify the PR title and description accordingly. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22708: [SPARK-21402] Fix java array/map of structs deser...

2018-10-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22708#discussion_r225391643 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -282,6 +283,27 @@ case class StaticInvoke

[GitHub] spark issue #22728: [SPARK-25736][SQL][TEST] add tests to verify the behavio...

2018-10-15 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22728 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22728: [SPARK-25736][SQL][TEST] add tests to verify the behavio...

2018-10-15 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22728 Yea, it is definitely good to add document and test for current behavior. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r225152625 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ddl.scala --- @@ -34,11 +34,16 @@ import org.apache.spark.sql.types

[GitHub] spark pull request #22713: [SPARK-25691][SQL] Use semantic equality in Alias...

2018-10-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22713#discussion_r225115756 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala --- @@ -112,8 +112,8 @@ object EliminateView extends Rule

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r225082659 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ddl.scala --- @@ -34,11 +34,16 @@ import org.apache.spark.sql.types

[GitHub] spark pull request #22662: [SPARK-25627][TEST] Reduce test time for Continuo...

2018-10-14 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22662#discussion_r224994803 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala --- @@ -259,10 +259,10 @@ class ContinuousStressSuite

[GitHub] spark issue #22655: [SPARK-25666][PYTHON] Internally document type conversio...

2018-10-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22655 @HyukjinKwon I can take some time to do similar for Pandas UDF. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22019: [WIP][SPARK-25040][SQL] Empty string for double and floa...

2018-10-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22019 @HyukjinKwon thanks for pinging me. Let's look at this and see if I can make a PR soon. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #22698: [SPARK-25710][SQL] range should report metrics correctly

2018-10-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22698 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22698: [SPARK-25710][SQL] range should report metrics co...

2018-10-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22698#discussion_r224659990 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -506,18 +513,18 @@ case class RangeExec(range

[GitHub] spark pull request #22698: [SPARK-25710][SQL] range should report metrics co...

2018-10-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22698#discussion_r224511158 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -506,18 +513,18 @@ case class RangeExec(range

[GitHub] spark issue #22696: [SPARK-25708][SQL] HAVING without GROUP BY means global ...

2018-10-11 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22696 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22696: [SPARK-25708][SQL] HAVING without GROUP BY means global ...

2018-10-11 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22696 I think we should mention this in migration guide. Although previous behavior is wrong, it might be treated as a "feature" of Spark SQL. We should explicitly let users know t

[GitHub] spark pull request #22696: [SPARK-25708][SQL] HAVING without GROUP BY means ...

2018-10-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22696#discussion_r224457317 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala --- @@ -108,7 +108,7 @@ class PlanParserSuite extends

[GitHub] spark issue #22688: [SPARK-25700][SQL] Creates ReadSupport in only Append Mo...

2018-10-11 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22688 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22688: [SPARK-25700][SQL] Creates ReadSupport in only Append Mo...

2018-10-11 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22688 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22688: [SPARK-25700][SQL] Creates ReadSupport in only Ap...

2018-10-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22688#discussion_r224337926 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/SimpleWritableDataSource.scala --- @@ -116,7 +116,6 @@ class SimpleWritableDataSource

<    1   2   3   4   5   6   7   8   9   10   >