[GitHub] spark pull request #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19181#discussion_r138735379 --- Diff: core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java --- @@ -503,6 +511,47 @@ public void

[GitHub] spark issue #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19181 cc @juliuszsompolski can you take a look, or perhaps take this for a test drive? --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19181#discussion_r138733992 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java --- @@ -162,14 +162,20 @@ private int

[GitHub] spark pull request #19181: [SPARK-21907][CORE] oom during spill

2017-09-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19181#discussion_r138733553 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java --- @@ -85,7 +85,7 @@ private final

[GitHub] spark issue #19181: [SPARK-21907][CORE] oom during spill

2017-09-12 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19181 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19175: [SPARK-21964][SQL]Enable splitting the Aggregate (on Exp...

2017-09-12 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19175 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19123: [SPARK-21418][SQL] NoSuchElementException: None.get in D...

2017-09-04 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19123 LGTM, merging to master/2.2. Thanks for picking this up! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19050: [SPARK-21835][SQL][WIP] RewritePredicateSubquery ...

2017-08-26 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19050#discussion_r135389263 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala --- @@ -98,6 +99,11 @@ object RewritePredicateSubquery

[GitHub] spark pull request #19050: [SPARK-21835][SQL][WIP] RewritePredicateSubquery ...

2017-08-25 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19050#discussion_r135271400 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala --- @@ -98,6 +99,11 @@ object RewritePredicateSubquery

[GitHub] spark pull request #19050: [SPARK-21835][SQL][WIP] RewritePredicateSubquery ...

2017-08-25 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19050#discussion_r135270779 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2502,3 +2373,140 @@ object UpdateOuterReferences

[GitHub] spark issue #19042: [SPARK-21830][SQL] Bump ANTLR version and fix a few issu...

2017-08-24 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19042 cc @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #19042: [SPARK-21830][SQL] Bump ANTLR version and fix a f...

2017-08-24 Thread hvanhovell
GitHub user hvanhovell opened a pull request: https://github.com/apache/spark/pull/19042 [SPARK-21830][SQL] Bump ANTLR version and fix a few issues. ## What changes were proposed in this pull request? This PR bumps the ANTLR version to 4.7, and fixes a number of small parser

[GitHub] spark issue #19036: [SPARK-21826][SQL] outer broadcast hash join should not ...

2017-08-24 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19036 Merging to master/2.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #19036: [SPARK-21826][SQL] outer broadcast hash join should not ...

2017-08-24 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19036 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #19010: [SPARK-21718][SQL] Heavy log of type: "Skipping partitio...

2017-08-21 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/19010 LGTM - merging to master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #18958: [SPARK-21745][SQL] Refactor ColumnVector hierarch...

2017-08-21 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18958#discussion_r134208734 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java --- @@ -0,0 +1,653 @@ +/* + * Licensed

[GitHub] spark issue #18993: [SPARK-21743][SQL][follow-up] top-most limit should not ...

2017-08-18 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18993 LGTM pending jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #18993: [SPARK-21743][SQL][follow-up] top-most limit shou...

2017-08-18 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18993#discussion_r133972522 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1180,6 +1180,9 @@ object ConvertToLocalRelation

[GitHub] spark pull request #18993: [SPARK-21743][SQL][follow-up] top-most limit shou...

2017-08-18 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18993#discussion_r133943823 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1180,6 +1180,9 @@ object ConvertToLocalRelation

[GitHub] spark issue #18959: [SPARK-18394][SQL] Make an AttributeSet.toSeq output ord...

2017-08-17 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18959 Merging to master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-08-17 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18979 cc @adrian-ionescu --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18955: [SPARK-21743][SQL] top-most limit should not cause memor...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18955 LGTM - pending jenkins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #18959: [SPARK-18394][SQL] Make an Attribute.toSeq output...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18959#discussion_r133445750 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeSet.scala --- @@ -121,7 +121,12 @@ class AttributeSet private

[GitHub] spark pull request #18955: [SPARK-21743][SQL] top-most limit should not caus...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18955#discussion_r133438086 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -72,7 +72,12 @@ abstract class SparkStrategies extends

[GitHub] spark issue #18958: [SPARK-21745][SQL] Refactor ColumnVector hierarchy to ma...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18958 On a more generic level. We could also choose to make `ColumnVectors` immutable, and create builder classes to create (reusable) instances; this would create a better separation between

[GitHub] spark pull request #18958: [SPARK-21745][SQL] Refactor ColumnVector hierarch...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18958#discussion_r133434818 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/MutableColumnVector.java --- @@ -0,0 +1,599 @@ +/* + * Licensed

[GitHub] spark pull request #18955: [SPARK-21743][SQL] top-most limit should not caus...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18955#discussion_r133422587 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala --- @@ -474,6 +474,10 @@ case class CollapseCodegenStages

[GitHub] spark pull request #18958: [SPARK-21745][SQL] Refactor ColumnVector hierarch...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18958#discussion_r133421918 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala --- @@ -89,14 +91,23 @@ class

[GitHub] spark pull request #18958: [SPARK-21745][SQL] Refactor ColumnVector hierarch...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18958#discussion_r133421832 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala --- @@ -89,14 +91,23 @@ class

[GitHub] spark pull request #18958: [SPARK-21745][SQL] Refactor ColumnVector hierarch...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18958#discussion_r133420801 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java --- @@ -491,6 +525,22 @@ public void loadBytes

[GitHub] spark pull request #18958: [SPARK-21745][SQL] Refactor ColumnVector hierarch...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18958#discussion_r133420728 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java --- @@ -40,8 +39,43 @@ private long

[GitHub] spark pull request #18958: [SPARK-21745][SQL] Refactor ColumnVector hierarch...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18958#discussion_r133419451 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/MutableColumnVector.java --- @@ -0,0 +1,599 @@ +/* + * Licensed

[GitHub] spark pull request #18958: [SPARK-21745][SQL] Refactor ColumnVector hierarch...

2017-08-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18958#discussion_r133418831 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnarBatch.java --- @@ -307,64 +293,73 @@ public void update(int ordinal

[GitHub] spark issue #18843: [SPARK-21595] Separate thresholds for buffering and spil...

2017-08-11 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18843 Merging to master/2.2. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18907: [SPARK-18464][SQL][followup] support old table which doe...

2017-08-11 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18907 @shaneknapp thanks for quick response! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18907: [SPARK-18464][SQL][followup] support old table which doe...

2017-08-11 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18907 I think something is up jenkins. @shaneknapp could you take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #18843: [SPARK-21595] Separate thresholds for buffering a...

2017-08-09 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18843#discussion_r132156490 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -844,24 +844,39 @@ object SQLConf { .stringConf

[GitHub] spark issue #18892: [SPARK-21520][SQL]Improvement a special case for non-det...

2017-08-09 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18892 @heary-cao can you improve the PR description, and explain what special cases you are improving. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #18843: [SPARK-21595] Separate thresholds for buffering and spil...

2017-08-09 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18843 LGTM - pending jenkins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18892: [SPARK-21520][SQL]Improvement a special case for non-det...

2017-08-09 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18892 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #18843: [SPARK-21595] Separate thresholds for buffering and spil...

2017-08-09 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18843 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18880: [SPARK-21665][Core]Need to close resources after use

2017-08-09 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18880 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #18836: Update SortMergeJoinExec.scala

2017-08-07 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18836#discussion_r131596479 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -82,7 +82,7 @@ case class SortMergeJoinExec

[GitHub] spark pull request #18854: [SPARK-21629][SQL][WIP] Fix Or nullability

2017-08-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18854#discussion_r131530967 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -396,6 +396,8 @@ case class Or(left: Expression

[GitHub] spark issue #18843: [SPARK-21595] Separate thresholds for buffering and spil...

2017-08-04 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18843 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.

2017-08-04 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18413 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18833: [SPARK-21625][SQL] sqrt(negative number) should be null.

2017-08-03 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18833 @maropu that only works for literals. I am sort-of in favor of the Hive default; it seems kinda bad to bring down a job because of negative value. --- If your project is set up for it, you can

[GitHub] spark pull request #18836: Update SortMergeJoinExec.scala

2017-08-03 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18836#discussion_r131239029 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -82,7 +82,7 @@ case class SortMergeJoinExec

[GitHub] spark issue #18679: [SPARK-21319][SQL] Fix memory leak in sorter

2017-07-27 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18679 LGTM - pending jenkins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-24 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r129005758 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala --- @@ -267,16 +267,17 @@ class

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-24 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r129005233 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala --- @@ -267,16 +267,17 @@ class

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-24 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r129005122 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -106,173 +101,167 @@ case class

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-24 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r129004992 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -106,173 +101,167 @@ case class

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-14 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r127453702 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -106,173 +105,164 @@ case class

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-14 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r127449331 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -106,173 +105,164 @@ case class

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-11 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r126680805 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -805,4 +806,24 @@ object TypeCoercion

[GitHub] spark pull request #18531: [SPARK-21307] [SQL] Remove SQLConf parameters fro...

2017-07-06 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18531#discussion_r125821262 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -45,11 +45,9 @@ import

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18543 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r125735576 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/WindowSpec.scala --- @@ -174,28 +191,22 @@ class WindowSpec private[sql

[GitHub] spark issue #18535: [SPARK-21312][SQL] correct offsetInBytes in UnsafeRow.wr...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18535 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18541 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18541#discussion_r125631732 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java --- @@ -588,6 +588,10 @@ public long getKeyPrefix

[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18541#discussion_r125631249 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java --- @@ -596,17 +600,37 @@ public long getKeyPrefix

[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18541#discussion_r125632002 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java --- @@ -596,17 +600,37 @@ public long getKeyPrefix

[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18541#discussion_r125631060 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java --- @@ -596,17 +600,37 @@ public long getKeyPrefix

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r125602833 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExec.scala --- @@ -109,46 +109,54 @@ case class WindowExec

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r125603187 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala --- @@ -151,6 +151,48 @@ class DataFrameWindowFunctionsSuite

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r125598577 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -43,57 +42,57 @@ case class

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r125599886 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -106,173 +105,161 @@ case class

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r125602934 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/WindowSpec.scala --- @@ -174,28 +191,22 @@ class WindowSpec private[sql

[GitHub] spark issue #18494: [SPARK-21272] SortMergeJoin LeftAnti does not update num...

2017-07-02 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18494 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #18483: [SPARK-17528][SQL] data should be copied properly...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18483#discussion_r125034337 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -1088,6 +1088,12 @@ public UTF8String clone

[GitHub] spark issue #18483: [SPARK-17528][SQL] data should be copied properly before...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18483 LGTM - pending jenkins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16699: [SPARK-18710][ML] Add offset in GLM

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16699 @yanboliang @actuaryzhang this PR breaks the scala-2.10 build: ``` [error] /home/jenkins/workspace/spark-master-compile-maven-scala-2.10/mllib/src/test/scala/org/apache/spark/ml

[GitHub] spark pull request #18483: [SPARK-17528][SQL] data should be copied properly...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18483#discussion_r125025743 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala --- @@ -94,4 +108,21 @@ object InternalRow

[GitHub] spark pull request #18483: [SPARK-17528][SQL] data should be copied properly...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18483#discussion_r125025770 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala --- @@ -94,4 +108,21 @@ object InternalRow

[GitHub] spark pull request #18483: [SPARK-17528][SQL] data should be copied properly...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18483#discussion_r124995524 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala --- @@ -94,4 +108,21 @@ object InternalRow

[GitHub] spark pull request #18483: [SPARK-17528][SQL] data should be copied properly...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18483#discussion_r124995580 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala --- @@ -94,4 +108,21 @@ object InternalRow

[GitHub] spark pull request #18483: [SPARK-17528][SQL] data should be copied properly...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18483#discussion_r124984978 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -1088,6 +1088,12 @@ public UTF8String clone

[GitHub] spark pull request #18483: [SPARK-17528][SQL] data should be copied properly...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18483#discussion_r124995623 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -1088,6 +1088,12 @@ public UTF8String clone

[GitHub] spark pull request #18483: [SPARK-17528][SQL] data should be copied properly...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18483#discussion_r124988718 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratedProjectionSuite.scala --- @@ -172,4 +172,40 @@ class

[GitHub] spark pull request #18483: [SPARK-17528][SQL] data should be copied properly...

2017-06-30 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18483#discussion_r124985506 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala --- @@ -94,4 +108,21 @@ object InternalRow

[GitHub] spark pull request #18470: [SPARK-21258][SQL] Fix WindowExec complex object ...

2017-06-29 Thread hvanhovell
GitHub user hvanhovell opened a pull request: https://github.com/apache/spark/pull/18470 [SPARK-21258][SQL] Fix WindowExec complex object aggregation with spilling ## What changes were proposed in this pull request? `WindowExec` currently improperly stores complex objects

[GitHub] spark issue #18440: [SPARK-21229][SQL] remove QueryPlan.preCanonicalized

2017-06-28 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18440 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #18412: [SPARK-21203] [SQL] Fix wrong results of insertio...

2017-06-26 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18412#discussion_r124066089 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala --- @@ -482,15 +482,15 @@ case class Cast(child: Expression

[GitHub] spark pull request #18412: [SPARK-21203] [SQL] Fix wrong results of insertio...

2017-06-24 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18412#discussion_r123876215 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala --- @@ -482,15 +482,15 @@ case class Cast(child: Expression

[GitHub] spark issue #18379: [SPARK-21164] [SQL] Remove isTableSample from Sample and...

2017-06-23 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18379 LGTM - pending jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...

2017-06-22 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16924 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #18387: [SPARK-21174] [SQL] Validate sampling fraction in...

2017-06-22 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18387#discussion_r123509528 --- Diff: sql/core/src/test/resources/sql-tests/results/tablesample-negative.sql.out --- @@ -0,0 +1,62 @@ +-- Automatically generated

[GitHub] spark issue #18379: [SPARK-21164] [SQL] Remove isTableSample from Sample

2017-06-21 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18379 Shouldn't we remove all the `isGenerated` stuff then? See `Alias` for example. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #18301: [SPARK-21052][SQL] Add hash map metrics to join

2017-06-14 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18301#discussion_r122116294 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala --- @@ -573,8 +586,11 @@ private[execution] final class

[GitHub] spark pull request #18301: [SPARK-21052][SQL] Add hash map metrics to join

2017-06-14 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/18301#discussion_r122114390 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala --- @@ -573,8 +586,11 @@ private[execution] final class

[GitHub] spark issue #17993: [SPARK-20758][SQL] Add Constant propagation optimization

2017-05-29 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/17993 LGTM - merging to master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17993: [SPARK-20758][SQL] Add Constant propagation optim...

2017-05-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/17993#discussion_r118852747 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala --- @@ -0,0 +1,167

[GitHub] spark pull request #17993: [SPARK-20758][SQL] Add Constant propagation optim...

2017-05-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/17993#discussion_r118852033 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala --- @@ -0,0 +1,167

[GitHub] spark pull request #17993: [SPARK-20758][SQL] Add Constant propagation optim...

2017-05-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/17993#discussion_r118852021 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala --- @@ -0,0 +1,167

[GitHub] spark pull request #17993: [SPARK-20758][SQL] Add Constant propagation optim...

2017-05-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/17993#discussion_r118851559 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -54,6 +54,62 @@ object ConstantFolding extends

[GitHub] spark pull request #17993: [SPARK-20758][SQL] Add Constant propagation optim...

2017-05-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/17993#discussion_r118851484 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -54,6 +54,62 @@ object ConstantFolding extends

[GitHub] spark pull request #17993: [SPARK-20758][SQL] Add Constant propagation optim...

2017-05-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/17993#discussion_r118851475 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -54,6 +54,62 @@ object ConstantFolding extends

[GitHub] spark pull request #17993: [SPARK-20758][SQL] Add Constant propagation optim...

2017-05-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/17993#discussion_r118846885 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -54,6 +54,62 @@ object ConstantFolding extends

<    2   3   4   5   6   7   8   9   10   11   >