[GitHub] spark issue #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21954 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207172497 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212 @@ +/* + * Licensed

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207169967 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212 @@ +/* + * Licensed

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207162371 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212 @@ +/* + * Licensed

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207162167 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212 @@ +/* + * Licensed

[GitHub] spark issue #21965: [WIP][SPARK-23909][SQL] Add filter function.

2018-08-02 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21965 This is based on #21954. I'll update after it is merged. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #21965: [WIP][SPARK-23909][SQL] Add filter function.

2018-08-02 Thread ueshin
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/21965 [WIP][SPARK-23909][SQL] Add filter function. ## What changes were proposed in this pull request? This pr adds `filter` function which filters the input array using the given predicate

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207148555 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212 @@ +/* + * Licensed

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207143138 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212 @@ +/* + * Licensed

[GitHub] spark issue #21958: [minor] remove dead code in ExpressionEvalHelper

2018-08-02 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21958 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21958: [minor] remove dead code in ExpressionEvalHelper

2018-08-02 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21958 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21954 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-01 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207102738 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,325 @@ +/* + * Licensed

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-01 Thread ueshin
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/21954 [SPARK-23908][SQL] Add transform function. ## What changes were proposed in this pull request? This pr adds `transform` function which transforms elements in an array using the function

[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.

2018-07-27 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21802 Thanks! merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21857: [SPARK-21274][SQL] Implement EXCEPT ALL clause.

2018-07-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21857#discussion_r205656881 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1400,13 +1401,71 @@ object

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r205650766 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,230 @@ object ArrayUnion

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r205649948 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,230 @@ object ArrayUnion

[GitHub] spark pull request #21802: [SPARK-23928][SQL] Add shuffle collection functio...

2018-07-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21802#discussion_r205534867 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RandomIndicesGenerator.scala --- @@ -0,0 +1,45 @@ +/* + * Licensed

[GitHub] spark pull request #21802: [SPARK-23928][SQL] Add shuffle collection functio...

2018-07-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21802#discussion_r205534846 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1184,6 +1184,110 @@ case class ArraySort

[GitHub] spark pull request #21857: [SPARK-21274][SQL] Implement EXCEPT ALL clause.

2018-07-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21857#discussion_r205357936 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1275,6 +1276,64 @@ object ReplaceExceptWithAntiJoin

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-26 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21103 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21102: [SPARK-23913][SQL] Add array_intersect function

2018-07-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21102#discussion_r205341581 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3801,339 @@ object ArrayUnion

[GitHub] spark pull request #21102: [SPARK-23913][SQL] Add array_intersect function

2018-07-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21102#discussion_r205342201 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3801,339 @@ object ArrayUnion

[GitHub] spark pull request #21102: [SPARK-23913][SQL] Add array_intersect function

2018-07-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21102#discussion_r205327639 --- Diff: core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala --- @@ -272,7 +272,7 @@ class OpenHashSet[@specialized(Long, Int) T

[GitHub] spark pull request #21830: [SPARK-24878][SQL] Fix reverse function for array...

2018-07-25 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21830#discussion_r205338930 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1244,46 +1244,50 @@ case class Reverse

[GitHub] spark pull request #21830: [SPARK-24878][SQL] Fix reverse function for array...

2018-07-25 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21830#discussion_r205338428 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1244,46 +1244,50 @@ case class Reverse

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-07-25 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21102 Just in my opinion, I'd prefer to preserve the element order of the left array. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-25 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r205186167 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,330 @@ object ArrayUnion

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-25 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r205186241 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,330 @@ object ArrayUnion

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-25 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r205186075 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,330 @@ object ArrayUnion

[GitHub] spark pull request #21850: [SPARK-24892] [SQL] Simplify `CaseWhen` to `If` w...

2018-07-25 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21850#discussion_r205022884 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -414,6 +414,16 @@ object SimplifyConditionals

[GitHub] spark issue #21869: [SPARK-24891][FOLLOWUP][HOT-FIX][2.3] Fix the Compilatio...

2018-07-25 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21869 LGTM, pending Jenkins. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21857: [SPARK-21274][SQL] Implement EXCEPT ALL clause.

2018-07-25 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21857#discussion_r204977789 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1275,6 +1276,64 @@ object ReplaceExceptWithAntiJoin

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-23 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r204307100 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,331 @@ object ArrayUnion

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-23 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r204306424 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3805,3 +3799,331 @@ object ArrayUnion

[GitHub] spark pull request #21802: [SPARK-23928][SQL] Add shuffle collection functio...

2018-07-22 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21802#discussion_r204275498 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -1419,4 +1421,71 @@ class

[GitHub] spark issue #21830: [SPARK-24878][SQL] Fix reverse function for array type o...

2018-07-20 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21830 cc @mn-mikke @cloud-fan @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark pull request #21830: [SPARK-24878][SQL] Fix reverse function for array...

2018-07-20 Thread ueshin
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/21830 [SPARK-24878][SQL] Fix reverse function for array type of primitive type containing null. ## What changes were proposed in this pull request? If we use `reverse` function for array type

[GitHub] spark pull request #21802: [SPARK-23928][SQL] Add shuffle collection functio...

2018-07-20 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21802#discussion_r203974038 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1184,6 +1186,137 @@ case class ArraySort

[GitHub] spark issue #21824: [SPARK-24871][SQL] Refactor Concat and MapConcat to avoi...

2018-07-20 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21824 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21802: [SPARK-23928][SQL] Add shuffle collection functio...

2018-07-20 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21802#discussion_r203962951 --- Diff: python/pyspark/sql/functions.py --- @@ -2382,6 +2382,20 @@ def array_sort(col): return Column(sc._jvm.functions.array_sort(_to_java_column

[GitHub] spark pull request #21802: [SPARK-23928][SQL] Add shuffle collection functio...

2018-07-20 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21802#discussion_r203963010 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2086,6 +2087,20 @@ class Analyzer

[GitHub] spark pull request #21802: [SPARK-23928][SQL] Add shuffle collection functio...

2018-07-20 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21802#discussion_r203963022 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1184,6 +1186,137 @@ case class ArraySort

[GitHub] spark issue #21824: [SPARK-24871][SQL] Refactor Concat and MapConcat to avoi...

2018-07-19 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21824 also cc @cloud-fan @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.

2018-07-19 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21802 cc @cloud-fan @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21824: [SPARK-24871][SQL] Refactor Concat and MapConcat to avoi...

2018-07-19 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21824 cc @mn-mikke @bersprockets --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21824: [SPARK-24871][SQL] Refactor Concat and MapConcat ...

2018-07-19 Thread ueshin
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/21824 [SPARK-24871][SQL] Refactor Concat and MapConcat to avoid creating concatenator object for each row. ## What changes were proposed in this pull request? Refactor `Concat` and `MapConcat

[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.

2018-07-19 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21802 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21386: [SPARK-23928][SQL][WIP] Add shuffle collection function.

2018-07-18 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21386 @pkuwm I submitted a PR #21802 based on this. Could you take a look if you have time? Thanks! --- - To unsubscribe, e-mail

[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.

2018-07-18 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21802 cc @pkuwm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21802: [SPARK-23928][SQL] Add shuffle collection functio...

2018-07-18 Thread ueshin
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/21802 [SPARK-23928][SQL] Add shuffle collection function. ## What changes were proposed in this pull request? This PR adds a new collection function: shuffle. It generates a random permutation

[GitHub] spark issue #21795: [SPARK-24840][SQL] do not use dummy filter to switch cod...

2018-07-18 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21795 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21102: [SPARK-23913][SQL] Add array_intersect function

2018-07-18 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21102#discussion_r203299689 --- Diff: core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala --- @@ -114,6 +118,21 @@ class OpenHashSet[@specialized(Long, Int) T

[GitHub] spark issue #21801: [SPARK-24386][SPARK-24768][BUILD][FOLLOWUP] Fix lint-jav...

2018-07-18 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21801 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21386: [SPARK-23928][SQL][WIP] Add shuffle collection function.

2018-07-17 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21386 Okay, I'll take this over, and ping you when I submit a PR to ask a review. Thanks! --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #21801: [SPARK-24386][SPARK-24768][BUILD][FOLLOWUP] Fix l...

2018-07-17 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21801#discussion_r203249250 --- Diff: external/avro/src/test/resources/log4j.properties --- @@ -46,4 +46,4 @@ log4j.additivity.org.apache.hadoop.hive.metastore.RetryingHMSHandler=false

[GitHub] spark pull request #21801: [SPARK-24386][SPARK-24768][BUILD][FOLLOWUP] Fix l...

2018-07-17 Thread ueshin
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/21801 [SPARK-24386][SPARK-24768][BUILD][FOLLOWUP] Fix lint-java and Scala 2.12 build. ## What changes were proposed in this pull request? This pr fixes lint-java and Scala 2.12 build

[GitHub] spark pull request #21795: [SPARK-24165][SQL][followup] Fixing conditional e...

2018-07-17 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21795#discussion_r203242468 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2336,46 +2336,40 @@ class DataFrameSuite extends QueryTest

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds arrays_zip function to sp...

2018-07-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r202693252 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -128,6 +128,172 @@ case class MapKeys

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds arrays_zip function to sp...

2018-07-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r202689850 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -128,6 +128,172 @@ case class MapKeys

[GitHub] spark pull request #21102: [SPARK-23913][SQL] Add array_intersect function

2018-07-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21102#discussion_r202621143 --- Diff: core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala --- @@ -73,6 +73,46 @@ class OpenHashSetSuite extends SparkFunSuite

[GitHub] spark issue #21704: [SPARK-24734][SQL] Fix type coercions and nullabilities ...

2018-07-16 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21704 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21386: [SPARK-23928][SQL][WIP] Add shuffle collection function.

2018-07-13 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21386 @pkuwm Hi, any updates on this? If you have any questions, please let us know. Thanks! --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #21704: [SPARK-24734][SQL] Fix type coercions and nullabi...

2018-07-13 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21704#discussion_r202278265 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -259,8 +270,22 @@ object TypeCoercion

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-07-12 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21061 Thanks! merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21745: [SPARK-24781][SQL] Using a reference from Dataset...

2018-07-12 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21745#discussion_r201930407 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1167,7 +1169,8 @@ class Analyzer( case p

[GitHub] spark pull request #21704: [SPARK-24734][SQL] Fix type coercions and nullabi...

2018-07-12 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21704#discussion_r201924130 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -610,27 +636,27 @@ object TypeCoercion

[GitHub] spark pull request #21704: [SPARK-24734][SQL] Fix containsNull of Concat for...

2018-07-12 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21704#discussion_r201919906 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -48,7 +48,8 @@ case class CreateArray

[GitHub] spark issue #21745: [SPARK-24781][SQL] Using a reference from Dataset in Fil...

2018-07-11 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21745 Actually, the very first time we introduce this bug was at 7463a88be6eecf14ba0a87048a975a06b03b928b. We added `!f.resolved &&` in `ResolveMissingReferenc

[GitHub] spark pull request #21704: [SPARK-24734][SQL] Fix containsNull of Concat for...

2018-07-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21704#discussion_r201900374 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -533,15 +559,15 @@ object TypeCoercion

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r201898932 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,322 @@ case class

[GitHub] spark pull request #21745: [SPARK-24781][SQL] Using a reference from Dataset...

2018-07-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21745#discussion_r201619815 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1163,7 +1163,8 @@ class Analyzer( case p

[GitHub] spark pull request #21745: [SPARK-24781][SQL] Using a reference from Dataset...

2018-07-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21745#discussion_r201621224 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/LogicalPlanSuite.scala --- @@ -119,4 +119,16 @@ class LogicalPlanSuite extends

[GitHub] spark pull request #21745: [SPARK-24781][SQL] Using a reference from Dataset...

2018-07-11 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21745#discussion_r201622249 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1163,7 +1163,8 @@ class Analyzer( case p

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-07-11 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21061 @viirya Do you have any other comments on this? Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21061: [SPARK-23914][SQL] Add array_union function

2018-07-10 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21061 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21687: [SPARK-24165][SQL] Fixing conditional expressions...

2018-07-10 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21687#discussion_r201264370 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -172,6 +172,18 @@ object TypeCoercion

[GitHub] spark pull request #21687: [SPARK-24165][SQL] Fixing conditional expressions...

2018-07-10 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21687#discussion_r201247091 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -685,10 +697,10 @@ object TypeCoercion

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-09 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21073 Thanks! merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21704: [SPARK-24734][SQL] Fix containsNull of Concat for array ...

2018-07-09 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21704 Note: we will be able to use `NonPrimitiveTypeMergingExpression` in #21687. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-09 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21073 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-09 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21073 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-09 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21073 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21687: [SPARK-24165][SQL] Fixing conditional expressions...

2018-07-08 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21687#discussion_r200886142 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala --- @@ -695,6 +695,56 @@ abstract class TernaryExpression

[GitHub] spark pull request #21687: [SPARK-24165][SQL] Fixing conditional expressions...

2018-07-08 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21687#discussion_r200879951 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala --- @@ -695,6 +695,56 @@ abstract class TernaryExpression

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-08 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200874571 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -1166,4 +1166,88 @@ class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-08 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200874190 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,323 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-08 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200874014 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,323 @@ case class

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-08 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21073 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-08 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21073 I'd retrigger the build, just in case. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-06 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21073 LGTM pending Jenkins. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-06 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21073 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200541043 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200543679 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200541138 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200540949 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200540913 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200541020 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21620#discussion_r200537707 --- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/arrayJoin.sql.out --- @@ -0,0 +1,90 @@ +-- Automatically generated

<    1   2   3   4   5   6   7   8   9   10   >