[GitHub] spark pull request #22468: [SPARK-25374][SQL] SafeProjection supports fallba...

2018-12-04 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22468#discussion_r238683833 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/UnsafeRowConverterSuite.scala --- @@ -535,4 +535,98 @@ class

[GitHub] spark issue #23127: [SPARK-26159] Codegen for LocalTableScanExec and RDDScan...

2018-11-23 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/23127 Looks good. One more higher level question that can also be addressed in a follow-up. --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #23127: [SPARK-26159] Codegen for LocalTableScanExec and ...

2018-11-23 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/23127#discussion_r236017398 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala --- @@ -406,14 +415,39 @@ trait

[GitHub] spark issue #23022: [SPARK-26038] Decimal toScalaBigInt/toJavaBigInteger for...

2018-11-23 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/23022 Merging to master. Thank! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #23096: [SPARK-26129][SQL] Instrumentation for per-query ...

2018-11-20 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/23096#discussion_r235159238 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -648,7 +648,11 @@ class SparkSession private( * @since 2.0.0

[GitHub] spark issue #23075: [SPARK-26084][SQL] Fixes unresolved AggregateExpression....

2018-11-20 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/23075 Also backported to 2.3/2.4. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #23075: [SPARK-26084][SQL] Fixes unresolved AggregateExpression....

2018-11-20 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/23075 Merging to master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #23075: [SPARK-26084][SQL] Fixes unresolved AggregateExpression....

2018-11-20 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/23075 Let's see if this works :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #23075: [SPARK-26084][SQL] Fixes unresolved AggregateExpression....

2018-11-20 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/23075 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23075: [SPARK-26084][SQL] Fixes unresolved AggregateExpression....

2018-11-19 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/23075 Ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23018: [SPARK-26023][SQL] Dumping truncated plans and generated...

2018-11-13 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/23018 Merging to master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #23018: [SPARK-26023][SQL] Dumping truncated plans and generated...

2018-11-13 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/23018 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22961: [SPARK-25947][SQL] Reduce memory usage in Shuffle...

2018-11-08 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22961#discussion_r232061457 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala --- @@ -214,13 +214,24 @@ object

[GitHub] spark issue #22964: [SPARK-25963] Optimize generate followed by window

2018-11-07 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22964 @uzadude where is this relevant? You will end up with two shuffles if you do this. --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22932: [SPARK-25102][SQL] Write Spark version to ORC/Par...

2018-11-04 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22932#discussion_r230604337 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/package.scala --- @@ -44,4 +44,13 @@ package object sql { type Strategy = SparkStrategy

[GitHub] spark issue #22925: [SPARK-25913][SQL] Extend UnaryExecNode by unary SparkPl...

2018-11-01 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22925 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22789: [SPARK-25767][SQL] Fix lazily evaluated stream of expres...

2018-10-29 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22789 Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22789: [SPARK-25767][SQL] Fix lazily evaluated stream of expres...

2018-10-29 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22789 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22789: [SPARK-25767][SQL] Fix lazily evaluated stream of...

2018-10-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22789#discussion_r228760802 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala --- @@ -319,4 +319,15 @@ class WholeStageCodegenSuite

[GitHub] spark pull request #22789: [SPARK-25767][SQL] Fix lazily evaluated stream of...

2018-10-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22789#discussion_r228749168 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala --- @@ -146,7 +146,10 @@ trait CodegenSupport extends

[GitHub] spark pull request #22789: [SPARK-25767][SQL] Fix lazily evaluated stream of...

2018-10-28 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22789#discussion_r228748979 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala --- @@ -319,4 +319,15 @@ class WholeStageCodegenSuite

[GitHub] spark issue #22789: [SPARK-25767][SQL] Fix lazily evaluated stream of expres...

2018-10-28 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22789 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22822: [SPARK-25678] Requesting feedback regarding a prototype ...

2018-10-25 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22822 @UtkarshMe well there is signal in the lack of responsiveness. Adding and maintaining cluster managers has proven to be quite painful, case and point is the lack of love that Mesos is receiving

[GitHub] spark issue #22822: [SPARK-25678] Requesting feedback regarding a prototype ...

2018-10-25 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22822 @UtkarshMe you should reach out to the spark dev list about this. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22817: [SPARK-25816][SQL] ResolveReferences should work bottom-...

2018-10-24 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22817 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22817: [SPARK-25816][SQL] ResolveReferences should work bottom-...

2018-10-24 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22817 @peter-toth what are you trying to fix here? Could you add this to the PR description? --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22576: [SPARK-25560][SQL] Allow FunctionInjection in Spa...

2018-10-19 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22576#discussion_r226623886 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala --- @@ -168,4 +173,21 @@ class SparkSessionExtensions { def

[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...

2018-10-19 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22576 @RussellSpitzer I am merging this, can you address my comment in a follow up? Thanks! --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22576: [SPARK-25560][SQL] Allow FunctionInjection in Spa...

2018-10-19 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22576#discussion_r226571338 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala --- @@ -168,4 +173,21 @@ class SparkSessionExtensions { def

[GitHub] spark pull request #22712: [SPARK-25724] Add sorting functionality in MapTyp...

2018-10-13 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22712#discussion_r224957118 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/MapType.scala --- @@ -73,6 +74,90 @@ case class MapType( override private[spark

[GitHub] spark pull request #22696: [SPARK-25708][SQL] HAVING without GROUP BY means ...

2018-10-11 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22696#discussion_r224590474 --- Diff: docs/sql-programming-guide.md --- @@ -1894,6 +1894,8 @@ working with timestamps in `pandas_udf`s to get the best performance, see

[GitHub] spark issue #22696: [SPARK-25708][SQL] HAVING without GROUP BY means global ...

2018-10-11 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22696 I added the release-notes label to the JIRA ticket. I am not sure if there is a migration-guide label. --- - To unsubscribe

[GitHub] spark pull request #22576: [SPARK-25560][SQL] Allow FunctionInjection in Spa...

2018-10-11 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22576#discussion_r224366907 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala --- @@ -168,4 +173,22 @@ class SparkSessionExtensions { def

[GitHub] spark pull request #22576: [SPARK-25560][SQL] Allow FunctionInjection in Spa...

2018-10-11 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22576#discussion_r224366774 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala --- @@ -95,7 +95,8 @@ abstract class

[GitHub] spark pull request #22576: [SPARK-25560][SQL] Allow FunctionInjection in Spa...

2018-10-11 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22576#discussion_r224364692 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala --- @@ -168,4 +173,22 @@ class SparkSessionExtensions { def

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-10-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r223983702 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala --- @@ -189,23 +192,34 @@ class QueryExecution(val sparkSession

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-10-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r223983537 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala --- @@ -167,6 +172,58 @@ package object util

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-10-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r223982046 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala --- @@ -167,6 +172,58 @@ package object util

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-10-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r223980665 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -455,21 +457,37 @@ abstract class TreeNode[BaseType

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-10-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r223979931 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/CatalystDataToAvro.scala --- @@ -52,7 +52,7 @@ case class CatalystDataToAvro(child

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-10-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r223979392 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -633,4 +633,14 @@ package object config { .stringConf

[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...

2018-10-10 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22429 @boy-uber the thing you are suggesting is a pretty big undertaking and beyond the scope of this PR. If you are going to add structured plans to the explain output, you probably also

[GitHub] spark pull request #22674: [SPARK-25680][SQL] SQL execution listener shouldn...

2018-10-09 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22674#discussion_r223886858 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala --- @@ -75,95 +76,74 @@ trait QueryExecutionListener

[GitHub] spark pull request #22674: [SPARK-25680][SQL] SQL execution listener shouldn...

2018-10-09 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22674#discussion_r223885742 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala --- @@ -75,95 +76,74 @@ trait QueryExecutionListener

[GitHub] spark pull request #22674: [SPARK-25680][SQL] SQL execution listener shouldn...

2018-10-09 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22674#discussion_r223873662 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala --- @@ -75,95 +76,74 @@ trait QueryExecutionListener

[GitHub] spark pull request #22674: [SPARK-25680][SQL] SQL execution listener shouldn...

2018-10-09 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22674#discussion_r223873406 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala --- @@ -39,7 +39,14 @@ case class SparkListenerSQLExecutionStart

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-09-18 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16677 1. `numOutputs` is the number or records 2. 8 bytes per `MapStatus`. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...

2018-09-18 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22429 @MaxGekk please just modify simpleString it is internal API for this reason. @rednaxelafx rope approach has the benefit that it does not create a ton of intermediate buffers. We could do

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r217928428 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala --- @@ -250,5 +254,36 @@ class QueryExecution(val sparkSession

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r217928334 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala --- @@ -250,5 +254,36 @@ class QueryExecution(val sparkSession

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r217928262 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -469,7 +470,17 @@ abstract class TreeNode[BaseType

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r217915071 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala --- @@ -250,5 +253,35 @@ class QueryExecution(val sparkSession

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-16 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r217913739 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -469,7 +470,13 @@ abstract class TreeNode[BaseType

[GitHub] spark pull request #22355: [SPARK-25358][SQL] MutableProjection supports fal...

2018-09-14 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22355#discussion_r217841164 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedMutableProjection.scala --- @@ -0,0 +1,83

[GitHub] spark issue #22417: [SPARK-25426][SQL] Remove the duplicate fallback logic i...

2018-09-14 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22417 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22344: [SPARK-25352][SQL] Perform ordered global limit w...

2018-09-12 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22344#discussion_r217070658 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -68,22 +68,42 @@ abstract class SparkStrategies extends

[GitHub] spark pull request #22205: [SPARK-25212][SQL] Support Filter in ConvertToLoc...

2018-08-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22205#discussion_r213124828 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1349,6 +1353,12 @@ object ConvertToLocalRelation

[GitHub] spark issue #22205: [SPARK-25212][SQL] Support Filter in ConvertToLocalRelat...

2018-08-27 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22205 @gatorsmile what are you afraid of exactly? We could check which tests are affected. Also do you want to disable this for testing only

[GitHub] spark issue #22239: [SPARK-19355][SQL][Followup] Remove the child.outputPart...

2018-08-26 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22239 Shall we rename it to: **[SPARK-19355][SQL][Followup] Remove the child.outputOrdering check in global limit

[GitHub] spark issue #22239: [SPARK-19355][SQL][Followup] Remove the child.outputPart...

2018-08-26 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22239 LGTM - Let's wait a little bit with merging to allow others to comment. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22239: [SPARK-19355][SQL][Followup] Remove the child.outputPart...

2018-08-26 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22239 Setting `spark.sql.limit.flatGlobalLimit` to `false` works for me. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22239: [SPARK-19355][SQL][Followup] Remove the child.outputPart...

2018-08-26 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22239 cc @cloud-fan for a sanity check. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22239: [SPARK-19355][SQL][Followup] Remove the child.outputPart...

2018-08-26 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22239 @viirya did you try to run `TakeOrderedAndProjectSuite`? I am pretty sure that will fail now ;)... --- - To unsubscribe, e

[GitHub] spark pull request #16677: [SPARK-19355][SQL] Use map output statistics to i...

2018-08-26 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16677#discussion_r212830045 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -93,25 +96,93 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark pull request #16677: [SPARK-19355][SQL] Use map output statistics to i...

2018-08-25 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16677#discussion_r212805707 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -93,25 +96,93 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark pull request #16677: [SPARK-19355][SQL] Use map output statistics to i...

2018-08-25 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16677#discussion_r212805327 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -93,25 +96,93 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark issue #22216: [SPARK-25223][SQL] Use a map to store values for NamedLa...

2018-08-24 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22216 I think the use of global state and a thread local is far more hacky and probably is slower. The only clean solution I see here is to pass the lambda values around using the input row

[GitHub] spark pull request #21899: [SPARK-24912][SQL] Don't obscure source of OOM du...

2018-08-17 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21899#discussion_r211044133 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala --- @@ -118,12 +119,20 @@ case class

[GitHub] spark pull request #22062: [SPARK-25081][Core]Nested spill in ShuffleExterna...

2018-08-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22062#discussion_r209372979 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/ShuffleExternalSorterSuite.scala --- @@ -0,0 +1,111 @@ +/* + * Licensed

[GitHub] spark pull request #14083: [SPARK-16406][SQL] Improve performance of Logical...

2018-08-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/14083#discussion_r209336169 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala --- @@ -138,6 +140,88 @@ package object expressions

[GitHub] spark pull request #22062: [SPARK-25081][Core]Nested spill in ShuffleExterna...

2018-08-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22062#discussion_r209292284 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/ShuffleExternalSorterSuite.scala --- @@ -0,0 +1,111 @@ +/* + * Licensed

[GitHub] spark pull request #22062: [SPARK-25081][Core]Nested spill in ShuffleExterna...

2018-08-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22062#discussion_r209291439 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/ShuffleExternalSorterSuite.scala --- @@ -0,0 +1,111 @@ +/* + * Licensed

[GitHub] spark pull request #22062: [SPARK-25081][Core]Nested spill in ShuffleExterna...

2018-08-10 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22062#discussion_r209262151 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java --- @@ -94,12 +94,20 @@ public int numRecords

[GitHub] spark issue #21369: [SPARK-22713][CORE] ExternalAppendOnlyMap leaks when spi...

2018-08-10 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21369 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22064: [MINOR][BUILD] Add ECCN notice required by http://www.ap...

2018-08-10 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22064 LGTM FWIW --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-08-10 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16677 Merging to master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22057: [SPARK-25077][SQL] Delete unused variable in WindowExec

2018-08-09 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22057 So I guess LGTM. I am generally not a fan of these aesthetic changes. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-08-08 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16677 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22035: [SPARK-23911][SQL][FOLLOW-UP] Fix examples of aggregate ...

2018-08-08 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22035 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-07 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r208273712 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/higherOrderFunctions.scala --- @@ -0,0 +1,166 @@ +/* + * Licensed

[GitHub] spark pull request #21258: [SPARK-23933][SQL] Add map_from_arrays function

2018-08-07 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21258#discussion_r208199133 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -235,6 +235,69 @@ case class CreateMap

[GitHub] spark pull request #22013: [SPARK-23939][SQL] Add transform_keys function

2018-08-07 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/22013#discussion_r208136330 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -365,3 +365,69 @@ case class

[GitHub] spark issue #22013: [SPARK-23939][SQL] Add transform_keys function

2018-08-06 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22013 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22012: [SPARK-25036][SQL] Should compare ExprValue.isNull with ...

2018-08-06 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/22012 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21986#discussion_r207954320 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -123,7 +125,10 @@ trait

[GitHub] spark issue #21982: [SPARK-23911][SQL] Add aggregate function.

2018-08-03 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21982 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21965: [SPARK-23909][SQL] Add filter function.

2018-08-03 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21965#discussion_r207480086 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -210,3 +219,54 @@ case class

[GitHub] spark pull request #21965: [SPARK-23909][SQL] Add filter function.

2018-08-03 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21965#discussion_r207479758 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -210,3 +219,54 @@ case class

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207171941 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207158636 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207145478 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212

[GitHub] spark pull request #21954: [SPARK-23908][SQL] Add transform function.

2018-08-02 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21954#discussion_r207141916 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -0,0 +1,212

[GitHub] spark issue #21930: [SPARK-14540][Core] Fix remaining major issues for Scala...

2018-07-31 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21930 Yeah I would not worry about it --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #21930: [SPARK-14540][Core] Fix remaining major issues fo...

2018-07-31 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21930#discussion_r206540368 --- Diff: core/src/main/scala/org/apache/spark/TaskContext.scala --- @@ -123,7 +123,7 @@ abstract class TaskContext extends Serializable

[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function

2018-07-29 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21103#discussion_r205967625 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3968,3 +3964,242 @@ object

[GitHub] spark issue #21897: [minor] Improve documentation for HiveStringType's

2018-07-27 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21897 @rxin no, they should not have been public. IMO we should just hide them for 3.0. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #21897: [minor] Improve documentation for HiveStringType's

2018-07-27 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21897 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21821: [SPARK-24867] [SQL] Add AnalysisBarrier to DataFrameWrit...

2018-07-25 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21821 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21840: [WIP] New copy() method for Column of StructType

2018-07-23 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/21840#discussion_r204476440 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3858,3 +3858,29 @@ object ArrayUnion

[GitHub] spark issue #21821: [SPARK-24867] [SQL] Add AnalysisBarrier to DataFrameWrit...

2018-07-23 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21821 @gatorsmile do we still need this patch if maryann fixes this? --- - To unsubscribe, e-mail: reviews-unsubscr

  1   2   3   4   5   6   7   8   9   10   >