Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23207
```var writer: ShuffleWriter[Any, Any] = null
try {
val manager = SparkEnv.get.shuffleManager
writer = manager.getWriter[Any, Any](
dep.shuffleHandle
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308829
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -170,13 +172,23 @@ class SQLMetricsSuite extends
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308706
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -95,3 +96,59 @@ private[spark] object
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308197
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -95,3 +96,59 @@ private[spark] object
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308082
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -38,12 +38,18 @@ case class CollectLimitExec(limit: Int, child:
SparkPlan
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308007
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -38,12 +38,18 @@ case class CollectLimitExec(limit: Int, child:
SparkPlan
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23207
@xuanyuanking can you separate the prs to rename read side metric and the
write side change?
---
-
To unsubscribe, e-mail: reviews
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238845399
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -299,12 +312,25 @@ class SQLMetricsSuite extends
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238845029
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -170,13 +172,23 @@ class SQLMetricsSuite extends
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238843017
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -163,6 +171,8 @@ object SQLMetrics
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238842276
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -78,6 +78,7 @@ object SQLMetrics {
private val
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238837000
--- Diff: core/src/main/scala/org/apache/spark/shuffle/metrics.scala ---
@@ -50,3 +50,57 @@ private[spark] trait ShuffleWriteMetricsReporter {
private
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238836448
--- Diff: core/src/main/scala/org/apache/spark/shuffle/metrics.scala ---
@@ -50,3 +50,57 @@ private[spark] trait ShuffleWriteMetricsReporter {
private
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23171
Basically logically there are only two expressions: In which handles
arbitrary expressions, and InSet which handles expressions with literals. Both
could work: (1) we provide two separate expressions
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23171
I thought InSwitch logically is the same as InSet, in which all the child
expressions are literals?
On Mon, Dec 03, 2018 at 8:38 PM, Wenchen Fan < notificati...@github.com >
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23171
That probably means we should just optimize InSet to have the switch
version though? Rather than do it in In?
On Mon, Dec 03, 2018 at 8:20 PM, Wenchen Fan < notificati...@github.com >
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23171
I'm not a big fan of making the physical implementation of an expression
very different depending on the situation. Why can't we just make InSet
efficient and convert these cases
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23192
Thanks @HyukjinKwon. Fixed it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23193
[SPARK-26226][SQL] Track optimization phase for streaming queries
## What changes were proposed in this pull request?
In an earlier PR, we missed measuring the optimization phase time
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23193
cc @gatorsmile @jose-torres
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23192
cc @zsxwing @jose-torres
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23192
[SPARK-26221][SQL] Add queryId to IncrementalExecution
## What changes were proposed in this pull request?
This is a small change for better debugging: to pass query uuid in
IncrementalExecution
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23183#discussion_r238019351
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/QueryPlanningTracker.scala
---
@@ -51,6 +58,18 @@ object QueryPlanningTracker
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23183
cc @hvanhovell @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23183
[SPARK-26226][SQL] Update query tracker to report timeline for phases
## What changes were proposed in this pull request?
This patch changes the query plan tracker added earlier to report phase
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23175
LGTM - merged in master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23178
Good idea to have it sealed!
> On Nov 29, 2018, at 7:04 AM, Sean Owen wrote:
>
> @srowen commented on this pull request.
>
> In
sql/core/src/main/scala/org/a
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23128
@xuanyuanking @cloud-fan when you think about where to put each code block,
make sure you also think about future evolution of the codebase. In general put
relevant things closer to each other (e.g
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r237129249
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -82,6 +82,14 @@ object SQLMetrics {
private val
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r237128247
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -0,0 +1,67 @@
+/*
+ * Licensed
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r237128189
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -194,4 +202,16 @@ object SQLMetrics
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23086#discussion_r236845375
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -38,7 +38,7 @@ import org.apache.spark.sql.execution.datasources.jdbc
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23106
Merging in master. Thanks @squito.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23086#discussion_r236492408
--- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/Table.java
---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23106#discussion_r236432889
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -242,8 +243,13 @@ private void writeSortedFile(boolean isLastFile
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23147
cc @gatorsmile @xuanyuanking
@cloud-fan I misunderstood your comment. Finally saw it today when I was
looking at my other PR
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23147
[SPARK-26140] followup: rename ShuffleMetricsReporter
## What changes were proposed in this pull request?
In https://github.com/apache/spark/pull/23105, due to working on two
parallel PRs at once
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23135#discussion_r236089467
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
---
@@ -575,6 +575,19 @@ case class Range
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23131#discussion_r236052557
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -1852,6 +1852,19 @@ class Dataset[T] private[sql](
CombineUnions(Union
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23129
Jenkins, test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236025838
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -0,0 +1,60 @@
+/*
+ * Licensed
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236025817
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -0,0 +1,60 @@
+/*
+ * Licensed
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23105#discussion_r236020103
--- Diff: core/src/main/scala/org/apache/spark/shuffle/metrics.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23105#discussion_r235950427
--- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala
---
@@ -48,7 +48,8 @@ private[spark] trait ShuffleManager {
handle
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23110
cc @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23110
[SPARK-26129] Followup - edge behavior for
QueryPlanningTracker.topRulesByTime
## What changes were proposed in this pull request?
This is an addendum patch for SPARK-26129 that defines the edge
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23106
[SPARK-26141] Enable custom shuffle metrics implementation in shuffle write
## What changes were proposed in this pull request?
This is the write side counterpart to
https://github.com/apache
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23105
cc @jiangxb1987 @squito
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23096
Merging this. Feel free to leave more comments. I'm hoping we can wire this
into the UI eventually.
---
-
To unsubscribe, e-mail
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23105#discussion_r235420647
--- Diff:
core/src/main/scala/org/apache/spark/executor/ShuffleReadMetrics.scala ---
@@ -122,34 +123,3 @@ class ShuffleReadMetrics private[spark] () extends
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23105
[SPARK-26140] Pull TempShuffleReadMetrics creation out of shuffle reader
## What changes were proposed in this pull request?
This patch defines an internal Spark interface for reporting shuffle
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235309483
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
---
@@ -648,7 +648,11 @@ class SparkSession private(
* @since 2.0.0
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23100
Change of this type can really piss some people off. Was there consensus on
this?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235182105
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala
---
@@ -88,15 +101,20 @@ abstract class RuleExecutor[TreeType
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235162047
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala
---
@@ -88,15 +92,18 @@ abstract class RuleExecutor[TreeType
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235161825
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -696,7 +701,7 @@ class Analyzer(
s
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235161336
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/QueryPlanningTracker.scala
---
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23096
cc @hvanhovell @gatorsmile
This is different from the existing metrics for rules as it is query
specific. We might want to replace that one with this in the future
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23096
[SPARK-26129][SQL] Instrumentation for query planning time
## What changes were proposed in this pull request?
We currently don't have good visibility into query planning time (analysis
vs
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23054#discussion_r234569150
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1594,6 +1594,15 @@ object SQLConf {
"WHERE, which
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23054
BTW what does the non-primitive types look like? Do they get flattened, or
is there a strict?
---
-
To unsubscribe, e-mail
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23054
We should add a âlegacyâ flag in case somebodyâs workload gets broken
by this. We can remove the legacy flag in a future release
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18784
Go for it.
On Fri, Nov 16, 2018 at 6:08 AM Stavros Kontopoulos <
notificati...@github.com> wrote:
> @imaxxs <https://github.com/imaxxs> @rxin <https://
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23021
One thing - I would put âpandasâ right after test_ so you get the
natural
logical grouping with sorting by file name.
On Tue, Nov 13, 2018 at 4:58 PM Hyukjin Kwon
wrote
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23021
Great initiative!
I'd break the pandas udf one into smaller pieces too, as you suggested. We
should also investigate why the runtime didn't improve
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22957
i didn't look at your new code, but is your old code safe? e.g. a project
that depends on the new alias.
---
-
To unsubscribe, e
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15899
Thanks for the example. I didn't even know that was possible in earlier
versions. I just looked it up: looks like Scala 2.11 rewrites for
comprehensions into map, filter, and flatMap
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/15899#discussion_r231390266
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -387,6 +387,14 @@ abstract class RDD[T: ClassTag](
preservesPartitioning = true
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22889
Yea good idea (prefer Array over Seq for short lists)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22921
seems good to me; might want to leave this open for a few days so more
people can take a look
---
-
To unsubscribe, e-mail
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22921#discussion_r230135473
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -62,17 +62,6 @@ class SQLContext private[sql](val sparkSession:
SparkSession
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22921#discussion_r230132632
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -639,20 +639,6 @@ private[spark] object SparkConf extends Logging
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22830
Perhaps @jkbradley and @mengxr can comment on it. If the trait is
inheritable, then protected still means it is part of the API contract
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22830
Who introduced this? We should ask the person that introduced it whether it
can be removed.
---
-
To unsubscribe, e-mail: reviews
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/22870
[SPARK-25862][SQL] Remove rangeBetween APIs introduced in SPARK-21608
## What changes were proposed in this pull request?
This patch removes the rangeBetween functions introduced in SPARK-21608
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22853#discussion_r228608016
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFramesSuite.scala
---
@@ -267,6 +267,25 @@ class DataFrameWindowFramesSuite extends
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22815#discussion_r228594291
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -54,6 +54,7 @@ import org.apache.spark.sql.util.ExecutionListenerManager
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21588
Does this upgrade Hive for execution or also for metastore? Spark supports
virtually all Hive metastore versions out there, and a lot of deployments do
run different versions of Spark against the same
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22841#discussion_r228376622
--- Diff: python/pyspark/sql/window.py ---
@@ -239,34 +212,27 @@ def rangeBetween(self, start, end):
and "5" means the five off after t
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22775#discussion_r228372331
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -770,8 +776,17 @@ case class SchemaOfJson
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22775
I agree it should be a literal value.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/22841
[SPARK-25842][SQL] Deprecate rangeBetween APIs introduced in SPARK-21608
## What changes were proposed in this pull request?
See the detailed information at
https://issues.apache.org/jira/browse
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22821
We seem to be splitting hairs here. Why are we providing tech preview to
advanced users? Are you saying they construct expressions directly using
internal APIs? I doubt thatâs tech preview
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22815
LGTM.
On a related note, we should probably deprecate the entire SQLContext.
---
-
To unsubscribe, e-mail: reviews
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22144
@markhamstra how did you arrive at that conclusion? I said "itâs not a
new regression and also
somewhat eso
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22144
Itâs certainly not a blocker since itâs not a new regression and also
somewhat esoteric. Would be good to fix though.
On Tue, Oct 23, 2018 at 8:20 AM Wenchen Fan
wrote
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21157
But that would break both ipython notebooks and repl right? Pretty
significant breaking change.
---
-
To unsubscribe, e-mail
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22010
If this is not yet in 2.4 it shouldnât be merged now.
On Wed, Oct 10, 2018 at 10:57 AM Holden Karau
wrote:
> Open question: is this suitable for branch-2.4 since it preda
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21157
@superbobry which blog were you referring to?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21157
so this change would introduce a pretty big regression?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22543#discussion_r220410457
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
---
@@ -1018,9 +1018,20 @@ case class TimeAdd(start
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22521
seems like our tests are really flaky
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22521
yup; just did
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/22541
[SPARK-23907][SQL] Revert regr_* functions entirely
## What changes were proposed in this pull request?
This patch reverts entirely all the regr_* functions added in SPARK-23907.
These were added
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22521
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/22521
[SPARK-24519] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_COMPRESS only once -
WIP
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21527#discussion_r219559889
--- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ---
@@ -50,7 +50,9 @@ private[spark] sealed trait MapStatus {
private[spark
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/22515
[SPARK-19724][SQL] allowCreatingManagedTableUsingNonemptyLocation should
have legacy prefix
One more legacy config to go ...
You can merge this pull request into a Git repository by running
Github user rxin closed the pull request at:
https://github.com/apache/spark/pull/22456
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22509
cc @dongjoon-hyun @MaxGekk we still need this pr don't we?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
1 - 100 of 14826 matches
Mail list logo
