This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 1fd7f29 [SPARK-28857][INFRA] Clean up the comments of PR template
during merging
add aef7ca1 [SPARK-28836
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from bdef712 [SPARK-28540][WEBUI] Document Environment page
add 4dc3093 [SPARK-28715][SQL] Introduce
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from a59fdc4 [SPARK-28472][SQL][TEST] Add test for thriftserver protocol
versions
add 325bc8e [SPARK-28583
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from f74ad3d [SPARK-28129][SQL][TEST] Port float8.sql
add 113f62d [SPARK-27485][FOLLOWUP] Do not reduce
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from e83583e [MINOR][SQL] Clean up ObjectProducerExec operators
add d1ef6be [SPARK-26978][SQL][FOLLOWUP
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 3663dbe [SPARK-28218][SQL] Migrate Avro to File Data Source V2
add e299f62 [SPARK-28241][SQL] Show
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c79f471 [SPARK-23128][SQL] A new approach
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from b508eab [SPARK-21882][CORE] OutputMetrics doesn't count written bytes
correctly in the saveAsHadoopDataset
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 00a8c85 [SPARK-27071][CORE] Expose
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new bd027f6 [SPARK-26656][SQL] Benchmarks
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9813b1d [SPARK-26690] Track query execution
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new e8e9b11 [SPARK-26680][SQL] Eagerly
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d4a30fa [SPARK-26680][SQL] Eagerly create
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 64ce1c9 [SPARK-26657][SQL] Use Proleptic
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 34db5f5 [SPARK-26618][SQL] Make typed
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6f8c0e5 [SPARK-26593][SQL] Use Proleptic
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 190814e [SPARK-26550][SQL] New built
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 09b0548 [SPARK-26450][SQL] Avoid rebuilding
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2a30deb [SPARK-26502][SQL] Move
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c036836 [SPARK-26495][SQL] Simplify
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 89c92cc [SPARK-26504][SQL] Rope-wise
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a1c1dd3 [SPARK-26191][SQL] Control
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22468#discussion_r238683833
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/UnsafeRowConverterSuite.scala
---
@@ -535,4 +535,98 @@ class
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/23127
Looks good. One more higher level question that can also be addressed in a
follow-up.
---
-
To unsubscribe, e-mail: reviews
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/23127#discussion_r236017398
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
---
@@ -406,14 +415,39 @@ trait
Repository: spark
Updated Branches:
refs/heads/master 8e8d1177e -> ecb785f4e
[SPARK-26038] Decimal toScalaBigInt/toJavaBigInteger for decimals not fitting
in long
## What changes were proposed in this pull request?
Fix Decimal `toScalaBigInt` and `toJavaBigInteger` used to only work for
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/23022
Merging to master. Thank!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235159238
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
---
@@ -648,7 +648,11 @@ class SparkSession private(
* @since 2.0.0
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/23075
Also backported to 2.3/2.4.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
ved expressions.
The refactored implementation is both simpler and faster, eliminating the
conversion of a `Set` to a
`Seq` and back to `Set`.
## How was this patch tested?
Added a new test based on the failing case in
[SPARK-26084](https://issues.apache.org/jira/browse/SPARK-26084).
hvanhov
ved expressions.
The refactored implementation is both simpler and faster, eliminating the
conversion of a `Set` to a
`Seq` and back to `Set`.
## How was this patch tested?
Added a new test based on the failing case in
[SPARK-26084](https://issues.apache.org/jira/browse/SPARK-26084).
hvanhov
ved expressions.
The refactored implementation is both simpler and faster, eliminating the
conversion of a `Set` to a
`Seq` and back to `Set`.
## How was this patch tested?
Added a new test based on the failing case in
[SPARK-26084](https://issues.apache.org/jira/browse/SPARK-26084).
hvanhovell
Clo
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/23075
Merging to master. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/23075
Let's see if this works :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/23075
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/23075
Ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Repository: spark
Updated Branches:
refs/heads/master c00e72f3d -> 44683e0f7
[SPARK-26023][SQL] Dumping truncated plans and generated code to a file
## What changes were proposed in this pull request?
In the PR, I propose new method for debugging queries by dumping info about
their
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/23018
Merging to master. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/23018
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22961#discussion_r232061457
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -214,13 +214,24 @@ object
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22964
@uzadude where is this relevant? You will end up with two shuffles if you
do this.
---
-
To unsubscribe, e-mail: reviews
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22932#discussion_r230604337
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/package.scala ---
@@ -44,4 +44,13 @@ package object sql {
type Strategy = SparkStrategy
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22925
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22789
Merged to master/2.4
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Repository: spark
Updated Branches:
refs/heads/branch-2.4 22bec3c6d -> 5cc2987db
[SPARK-25767][SQL] Fix lazily evaluated stream of expressions in code generation
## What changes were proposed in this pull request?
Code generation is incorrect if `outputVars` parameter of `consume` method in
Repository: spark
Updated Branches:
refs/heads/master 409d688fb -> 7fe5cff05
[SPARK-25767][SQL] Fix lazily evaluated stream of expressions in code generation
## What changes were proposed in this pull request?
Code generation is incorrect if `outputVars` parameter of `consume` method in
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22789
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22789#discussion_r228760802
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
---
@@ -319,4 +319,15 @@ class WholeStageCodegenSuite
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22789#discussion_r228749168
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
---
@@ -146,7 +146,10 @@ trait CodegenSupport extends
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22789#discussion_r228748979
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
---
@@ -319,4 +319,15 @@ class WholeStageCodegenSuite
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22789
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22822
@UtkarshMe well there is signal in the lack of responsiveness. Adding and
maintaining cluster managers has proven to be quite painful, case and point is
the lack of love that Mesos is receiving
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22822
@UtkarshMe you should reach out to the spark dev list about this.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22817
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22817
@peter-toth what are you trying to fix here? Could you add this to the PR
description?
---
-
To unsubscribe, e-mail: reviews
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22576#discussion_r226623886
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala ---
@@ -168,4 +173,21 @@ class SparkSessionExtensions {
def
Repository: spark
Updated Branches:
refs/heads/master c8f7691c6 -> 6e0fc8b0f
[SPARK-25560][SQL] Allow FunctionInjection in SparkExtensions
This allows an implementer of Spark Session Extensions to utilize a
method "injectFunction" which will add a new function to the default
Spark Session
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22576
@RussellSpitzer I am merging this, can you address my comment in a follow
up? Thanks!
---
-
To unsubscribe, e-mail: reviews
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22576#discussion_r226571338
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala ---
@@ -168,4 +173,21 @@ class SparkSessionExtensions {
def
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22712#discussion_r224957118
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/MapType.scala ---
@@ -73,6 +74,90 @@ case class MapType(
override private[spark
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22696#discussion_r224590474
--- Diff: docs/sql-programming-guide.md ---
@@ -1894,6 +1894,8 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22696
I added the release-notes label to the JIRA ticket. I am not sure if there
is a migration-guide label.
---
-
To unsubscribe
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22576#discussion_r224366907
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala ---
@@ -168,4 +173,22 @@ class SparkSessionExtensions {
def
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22576#discussion_r224366774
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala
---
@@ -95,7 +95,8 @@ abstract class
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22576#discussion_r224364692
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala ---
@@ -168,4 +173,22 @@ class SparkSessionExtensions {
def
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223983702
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ---
@@ -189,23 +192,34 @@ class QueryExecution(val sparkSession
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223983537
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala ---
@@ -167,6 +172,58 @@ package object util
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223982046
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala ---
@@ -167,6 +172,58 @@ package object util
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223980665
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
---
@@ -455,21 +457,37 @@ abstract class TreeNode[BaseType
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223979931
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/CatalystDataToAvro.scala
---
@@ -52,7 +52,7 @@ case class CatalystDataToAvro(child
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r223979392
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -633,4 +633,14 @@ package object config {
.stringConf
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22429
@boy-uber the thing you are suggesting is a pretty big undertaking and
beyond the scope of this PR.
If you are going to add structured plans to the explain output, you
probably also
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r223886858
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala
---
@@ -75,95 +76,74 @@ trait QueryExecutionListener
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r223885742
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala
---
@@ -75,95 +76,74 @@ trait QueryExecutionListener
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r223873662
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala
---
@@ -75,95 +76,74 @@ trait QueryExecutionListener
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r223873406
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala ---
@@ -39,7 +39,14 @@ case class SparkListenerSQLExecutionStart
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/16677
1. `numOutputs` is the number or records
2. 8 bytes per `MapStatus`.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22429
@MaxGekk please just modify simpleString it is internal API for this reason.
@rednaxelafx rope approach has the benefit that it does not create a ton of
intermediate buffers. We could do
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r217928428
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ---
@@ -250,5 +254,36 @@ class QueryExecution(val sparkSession
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r217928334
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ---
@@ -250,5 +254,36 @@ class QueryExecution(val sparkSession
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r217928262
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
---
@@ -469,7 +470,17 @@ abstract class TreeNode[BaseType
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r217915071
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ---
@@ -250,5 +253,35 @@ class QueryExecution(val sparkSession
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r217913739
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
---
@@ -469,7 +470,13 @@ abstract class TreeNode[BaseType
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22355#discussion_r217841164
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedMutableProjection.scala
---
@@ -0,0 +1,83
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22417
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Repository: spark
Updated Branches:
refs/heads/branch-2.4 abb5196c7 -> e7f511ad0
[SPARK-25352][SQL][FOLLOWUP] Add helper method and address style issue
## What changes were proposed in this pull request?
This follow-up patch addresses [the review
Repository: spark
Updated Branches:
refs/heads/master 3e75a9fa2 -> 5b761c537
[SPARK-25352][SQL][FOLLOWUP] Add helper method and address style issue
## What changes were proposed in this pull request?
This follow-up patch addresses [the review
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22344#discussion_r217070658
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -68,22 +68,42 @@ abstract class SparkStrategies extends
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22205#discussion_r213124828
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1349,6 +1353,12 @@ object ConvertToLocalRelation
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22205
@gatorsmile what are you afraid of exactly? We could check which tests are
affected. Also do you want to disable this for testing only
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22239
Shall we rename it to: **[SPARK-19355][SQL][Followup] Remove the
child.outputOrdering check in global limit
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22239
LGTM - Let's wait a little bit with merging to allow others to comment.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22239
Setting `spark.sql.limit.flatGlobalLimit` to `false` works for me.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22239
cc @cloud-fan for a sanity check.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22239
@viirya did you try to run `TakeOrderedAndProjectSuite`? I am pretty sure
that will fail now ;)...
---
-
To unsubscribe, e
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16677#discussion_r212830045
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -93,25 +96,93 @@ trait BaseLimitExec extends UnaryExecNode
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16677#discussion_r212805707
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -93,25 +96,93 @@ trait BaseLimitExec extends UnaryExecNode
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16677#discussion_r212805327
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -93,25 +96,93 @@ trait BaseLimitExec extends UnaryExecNode
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22216
I think the use of global state and a thread local is far more hacky and
probably is slower.
The only clean solution I see here is to pass the lambda values around
using the input row
Repository: spark
Updated Branches:
refs/heads/master b88ddb8a8 -> cd6dff78b
[SPARK-25209][SQL] Avoid deserializer check in Dataset.apply when Dataset is
actually DataFrame
## What changes were proposed in this pull request?
Dataset.apply calls dataset.deserializer (to provide an early
401 - 500 of 5056 matches
Mail list logo
