This is an automated email from the ASF dual-hosted git repository.
rxin pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new aa39b06462a [MINOR][TEST][SQL] Add a CTE
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 24edf8ecb5e [MINOR][TEST][SQL] Add a CTE subquery
Author: rxin
Date: Tue Mar 2 11:00:12 2021
New Revision: 46414
Log:
Moving Apache Spark 3.1.1 RC3 to Apache Spark 3.1.1
Added:
release/spark/spark-3.1.1/
- copied from r46413, dev/spark/v3.1.1-rc3-bin/
Removed:
dev/spark/v3.1.1-rc3-bin
Author: rxin
Date: Tue Mar 2 10:55:39 2021
New Revision: 46413
Log:
Recover 3.1.1 RC3
Added:
dev/spark/v3.1.1-rc3-bin/
- copied from r46410, dev/spark/v3.1.1-rc3-bin/
dev/spark/v3.1.1-rc3-docs/
- copied from r46410, dev/spark/v3.1.1-rc3-docs
Author: rxin
Date: Tue Mar 2 10:39:38 2021
New Revision: 46411
Log:
Removing RC artifacts.
Removed:
dev/spark/v3.1.1-rc3-bin/
dev/spark/v3.1.1-rc3-docs/
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
Author: rxin
Date: Tue Mar 2 10:39:58 2021
New Revision: 46412
Log:
Removing RC artifacts.
Removed:
dev/spark/v3.1.0-rc1-bin/
dev/spark/v3.1.0-rc1-docs/
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
Author: rxin
Date: Tue Mar 2 10:39:32 2021
New Revision: 46410
Log:
Removing RC artifacts.
Removed:
dev/spark/v3.1.1-rc2-bin/
dev/spark/v3.1.1-rc2-docs/
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
Author: rxin
Date: Tue Mar 2 10:39:25 2021
New Revision: 46409
Log:
Removing RC artifacts.
Removed:
dev/spark/v3.1.1-rc1-bin/
dev/spark/v3.1.1-rc1-docs/
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
Author: rxin
Date: Thu Jun 18 16:41:27 2020
New Revision: 40088
Log:
Removing RC artifacts.
Removed:
dev/spark/v3.0.0-rc1-bin/
dev/spark/v3.0.0-rc1-docs/
dev/spark/v3.0.0-rc2-bin/
dev/spark/v3.0.0-rc2-docs/
dev/spark/v3.0.0-rc3-docs
Author: rxin
Date: Tue Jun 16 09:18:02 2020
New Revision: 40050
Log:
release 3.0.0
Added:
release/spark/spark-3.0.0/
- copied from r40049, dev/spark/v3.0.0-rc3-bin/
Removed:
dev/spark/v3.0.0-rc3-bin
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a change to tag v3.0.0
in repository https://gitbox.apache.org/repos/asf/spark.git.
at 3fdfce3 (commit)
No new revisions were added by this update
Author: rxin
Date: Sat Jun 6 14:03:25 2020
New Revision: 39960
Log:
Apache Spark v3.0.0-rc3 docs
[This commit notification would consist of 1920 parts,
which exceeds the limit of 50 ones, so it was shortened to the summary
Author: rxin
Date: Sat Jun 6 13:35:40 2020
New Revision: 39959
Log:
Apache Spark v3.0.0-rc3
Added:
dev/spark/v3.0.0-rc3-bin/
dev/spark/v3.0.0-rc3-bin/SparkR_3.0.0.tar.gz (with props)
dev/spark/v3.0.0-rc3-bin/SparkR_3.0.0.tar.gz.asc
dev/spark/v3.0.0-rc3-bin/SparkR_3.0.0
Author: rxin
Date: Sat Jun 6 11:18:32 2020
New Revision: 39958
Log:
remove 3.0 rc3 binary
Removed:
dev/spark/v3.0.0-rc3-bin/
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git.
from fa608b9 [SPARK-31904][SQL] Fix case sensitive problem of char and
varchar partition columns
add 3fdfce3
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 3ea461d61e635835c07bacb5a0c403ae2a3099a0
Author: Reynold Xin
AuthorDate: Sat Jun 6 02:57:41 2020 +
Preparing
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a commit to tag v3.0.0-rc3
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 3fdfce3120f307147244e5eaf46d61419a723d50
Author: Reynold Xin
AuthorDate: Sat Jun 6 02:57:35 2020 +
Preparing
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a change to tag v3.0.0-rc3
in repository https://gitbox.apache.org/repos/asf/spark.git.
at 3fdfce3 (commit)
This tag includes the following new commits:
new 3fdfce3 Preparing Spark release v3.0.0-rc3
Author: rxin
Date: Fri Jun 5 19:08:09 2020
New Revision: 39951
Log:
Apache Spark v3.0.0-rc3
Added:
dev/spark/v3.0.0-rc3-bin/
dev/spark/v3.0.0-rc3-bin/SparkR_3.0.0.tar.gz (with props)
dev/spark/v3.0.0-rc3-bin/SparkR_3.0.0.tar.gz.asc
dev/spark/v3.0.0-rc3-bin/SparkR_3.0.0
Author: rxin
Date: Mon May 18 16:11:38 2020
New Revision: 39657
Log:
Apache Spark v3.0.0-rc2 docs
[This commit notification would consist of 1921 parts,
which exceeds the limit of 50 ones, so it was shortened to the summary
Author: rxin
Date: Mon May 18 15:42:56 2020
New Revision: 39656
Log:
Apache Spark v3.0.0-rc2
Added:
dev/spark/v3.0.0-rc2-bin/
dev/spark/v3.0.0-rc2-bin/SparkR_3.0.0.tar.gz (with props)
dev/spark/v3.0.0-rc2-bin/SparkR_3.0.0.tar.gz.asc
dev/spark/v3.0.0-rc2-bin/SparkR_3.0.0
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 740da34 [SPARK-31738][SQL][DOCS] Describe 'L' and 'M' month pattern
letters
add 29853ec Preparing Spark
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a commit to tag v3.0.0-rc2
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 29853eca69bceefd227cbe8421a09c116b7b753a
Author: Reynold Xin
AuthorDate: Mon May 18 13:21:37 2020 +
Preparing
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a change to tag v3.0.0-rc2
in repository https://gitbox.apache.org/repos/asf/spark.git.
at 29853ec (commit)
This tag includes the following new commits:
new 29853ec Preparing Spark release v3.0.0-rc2
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit f6053b94f874c62856baa7bfa35df14c78bebc9f
Author: Reynold Xin
AuthorDate: Mon May 18 13:21:43 2020 +
Preparing
Author: rxin
Date: Tue Mar 31 13:45:27 2020
New Revision: 38759
Log:
Apache Spark v3.0.0-rc1 docs
[This commit notification would consist of 1911 parts,
which exceeds the limit of 50 ones, so it was shortened to the summary
Author: rxin
Date: Tue Mar 31 09:57:10 2020
New Revision: 38754
Log:
Apache Spark v3.0.0-rc1
Added:
dev/spark/v3.0.0-rc1-bin/
dev/spark/v3.0.0-rc1-bin/SparkR_3.0.0.tar.gz (with props)
dev/spark/v3.0.0-rc1-bin/SparkR_3.0.0.tar.gz.asc
dev/spark/v3.0.0-rc1-bin/SparkR_3.0.0
Author: rxin
Date: Tue Mar 31 07:25:15 2020
New Revision: 38753
Log:
retry
Removed:
dev/spark/v3.0.0-rc1-bin/
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h
Author: rxin
Date: Mon Mar 30 16:00:46 2020
New Revision: 38740
Log:
Apache Spark v3.0.0-rc1
Added:
dev/spark/v3.0.0-rc1-bin/
dev/spark/v3.0.0-rc1-bin/SparkR_3.0.0.tar.gz (with props)
dev/spark/v3.0.0-rc1-bin/SparkR_3.0.0.tar.gz.asc
dev/spark/v3.0.0-rc1-bin/SparkR_3.0.0
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit fc5079841907443369af98b17c20f1ac24b3727d
Author: Reynold Xin
AuthorDate: Mon Mar 30 08:42:27 2020 +
Preparing
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 5687b31 [SPARK-30532] DataFrameStatFunctions to work with
TABLE.COLUMN syntax
add 6550d0d Preparing Spark
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a change to tag v3.0.0-rc1
in repository https://gitbox.apache.org/repos/asf/spark.git.
at 6550d0d (commit)
This tag includes the following new commits:
new 6550d0d Preparing Spark release v3.0.0-rc1
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a commit to tag v3.0.0-rc1
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 6550d0d5283efdbbd838f3aeaf0476c7f52a0fb1
Author: Reynold Xin
AuthorDate: Mon Mar 30 08:42:10 2020 +
Preparing
Author: rxin
Date: Mon Mar 30 07:26:00 2020
New Revision: 38725
Log:
Update KEYS
Modified:
dev/spark/KEYS
Modified: dev/spark/KEYS
==
--- dev/spark/KEYS (original)
+++ dev/spark/KEYS Mon Mar 30 07:26:00 2020
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a change to branch test-branch
in repository https://gitbox.apache.org/repos/asf/spark.git.
was 0f8b07e test
This change permanently discards the following revisions:
discard 0f8b07e test
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a change to branch test-branch
in repository https://gitbox.apache.org/repos/asf/spark.git.
at 0f8b07e test
This branch includes the following new commits:
new 0f8b07e test
The 1 revisions listed
This is an automated email from the ASF dual-hosted git repository.
rxin pushed a commit to branch test-branch
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 0f8b07e5034af2819b75b53aadffda82ae0c31b8
Author: Reynold Xin
AuthorDate: Fri Feb 1 13:28:18 2019 -0800
test
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23207
```var writer: ShuffleWriter[Any, Any] = null
try {
val manager = SparkEnv.get.shuffleManager
writer = manager.getWriter[Any, Any](
dep.shuffleHandle
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308829
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -170,13 +172,23 @@ class SQLMetricsSuite extends
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308706
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -95,3 +96,59 @@ private[spark] object
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308197
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -95,3 +96,59 @@ private[spark] object
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308082
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -38,12 +38,18 @@ case class CollectLimitExec(limit: Int, child:
SparkPlan
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r239308007
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala ---
@@ -38,12 +38,18 @@ case class CollectLimitExec(limit: Int, child:
SparkPlan
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23207
@xuanyuanking can you separate the prs to rename read side metric and the
write side change?
---
-
To unsubscribe, e-mail: reviews
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238845399
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -299,12 +312,25 @@ class SQLMetricsSuite extends
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238845029
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -170,13 +172,23 @@ class SQLMetricsSuite extends
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238843017
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -163,6 +171,8 @@ object SQLMetrics
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238842276
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -78,6 +78,7 @@ object SQLMetrics {
private val
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238837000
--- Diff: core/src/main/scala/org/apache/spark/shuffle/metrics.scala ---
@@ -50,3 +50,57 @@ private[spark] trait ShuffleWriteMetricsReporter {
private
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23207#discussion_r238836448
--- Diff: core/src/main/scala/org/apache/spark/shuffle/metrics.scala ---
@@ -50,3 +50,57 @@ private[spark] trait ShuffleWriteMetricsReporter {
private
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23171
Basically logically there are only two expressions: In which handles
arbitrary expressions, and InSet which handles expressions with literals. Both
could work: (1) we provide two separate expressions
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23171
I thought InSwitch logically is the same as InSet, in which all the child
expressions are literals?
On Mon, Dec 03, 2018 at 8:38 PM, Wenchen Fan < notificati...@github.com >
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23171
That probably means we should just optimize InSet to have the switch
version though? Rather than do it in In?
On Mon, Dec 03, 2018 at 8:20 PM, Wenchen Fan < notificati...@github.com >
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23171
I'm not a big fan of making the physical implementation of an expression
very different depending on the situation. Why can't we just make InSet
efficient and convert these cases
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23192
Thanks @HyukjinKwon. Fixed it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23193
[SPARK-26226][SQL] Track optimization phase for streaming queries
## What changes were proposed in this pull request?
In an earlier PR, we missed measuring the optimization phase time
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23193
cc @gatorsmile @jose-torres
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23192
cc @zsxwing @jose-torres
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23192
[SPARK-26221][SQL] Add queryId to IncrementalExecution
## What changes were proposed in this pull request?
This is a small change for better debugging: to pass query uuid in
IncrementalExecution
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23183#discussion_r238019351
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/QueryPlanningTracker.scala
---
@@ -51,6 +58,18 @@ object QueryPlanningTracker
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23183
cc @hvanhovell @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23183
[SPARK-26226][SQL] Update query tracker to report timeline for phases
## What changes were proposed in this pull request?
This patch changes the query plan tracker added earlier to report phase
Repository: spark
Updated Branches:
refs/heads/master 9fdc7a840 -> cb368f2c2
[SPARK-26142] followup: Move sql shuffle read metrics relatives to
SQLShuffleMetricsReporter
## What changes were proposed in this pull request?
Follow up for https://github.com/apache/spark/pull/23128, move sql
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23175
LGTM - merged in master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23178
Good idea to have it sealed!
> On Nov 29, 2018, at 7:04 AM, Sean Owen wrote:
>
> @srowen commented on this pull request.
>
> In
sql/core/src/main/scala/org/a
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23128
@xuanyuanking @cloud-fan when you think about where to put each code block,
make sure you also think about future evolution of the codebase. In general put
relevant things closer to each other (e.g
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r237129249
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -82,6 +82,14 @@ object SQLMetrics {
private val
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r237128247
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -0,0 +1,67 @@
+/*
+ * Licensed
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r237128189
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -194,4 +202,16 @@ object SQLMetrics
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23086#discussion_r236845375
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -38,7 +38,7 @@ import org.apache.spark.sql.execution.datasources.jdbc
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23106
Merging in master. Thanks @squito.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
How was this patch tested?
No behavior change expected, as it is a straightforward refactoring. Updated
all existing test cases.
Closes #23106 from rxin/SPARK-26141.
Authored-by: Reynold Xin
Signed-off-by: Reynold Xin
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23086#discussion_r236492408
--- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/Table.java
---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23106#discussion_r236432889
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -242,8 +243,13 @@ private void writeSortedFile(boolean isLastFile
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23147
cc @gatorsmile @xuanyuanking
@cloud-fan I misunderstood your comment. Finally saw it today when I was
looking at my other PR
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23147
[SPARK-26140] followup: rename ShuffleMetricsReporter
## What changes were proposed in this pull request?
In https://github.com/apache/spark/pull/23105, due to working on two
parallel PRs at once
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23135#discussion_r236089467
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
---
@@ -575,6 +575,19 @@ case class Range
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23131#discussion_r236052557
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -1852,6 +1852,19 @@ class Dataset[T] private[sql](
CombineUnions(Union
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23129
Jenkins, test this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236025838
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -0,0 +1,60 @@
+/*
+ * Licensed
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23128#discussion_r236025817
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala
---
@@ -0,0 +1,60 @@
+/*
+ * Licensed
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23105#discussion_r236020103
--- Diff: core/src/main/scala/org/apache/spark/shuffle/metrics.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23105#discussion_r235950427
--- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala
---
@@ -48,7 +48,8 @@ private[spark] trait ShuffleManager {
handle
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23110
cc @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23110
[SPARK-26129] Followup - edge behavior for
QueryPlanningTracker.topRulesByTime
## What changes were proposed in this pull request?
This is an addendum patch for SPARK-26129 that defines the edge
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23106
[SPARK-26141] Enable custom shuffle metrics implementation in shuffle write
## What changes were proposed in this pull request?
This is the write side counterpart to
https://github.com/apache
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23105
cc @jiangxb1987 @squito
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
vs physical planning). This patch adds a simple utility to track
the runtime of various rules and various planning phases.
## How was this patch tested?
Added unit tests and end-to-end integration tests.
Closes #23096 from rxin/SPARK-26129.
Authored-by: Reynold Xin
Signed-off-by: Reynold Xin
Proj
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23096
Merging this. Feel free to leave more comments. I'm hoping we can wire this
into the UI eventually.
---
-
To unsubscribe, e-mail
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23105#discussion_r235420647
--- Diff:
core/src/main/scala/org/apache/spark/executor/ShuffleReadMetrics.scala ---
@@ -122,34 +123,3 @@ class ShuffleReadMetrics private[spark] () extends
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23105
[SPARK-26140] Pull TempShuffleReadMetrics creation out of shuffle reader
## What changes were proposed in this pull request?
This patch defines an internal Spark interface for reporting shuffle
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235309483
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
---
@@ -648,7 +648,11 @@ class SparkSession private(
* @since 2.0.0
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23100
Change of this type can really piss some people off. Was there consensus on
this?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235182105
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala
---
@@ -88,15 +101,20 @@ abstract class RuleExecutor[TreeType
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235162047
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala
---
@@ -88,15 +92,18 @@ abstract class RuleExecutor[TreeType
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235161825
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -696,7 +701,7 @@ class Analyzer(
s
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23096#discussion_r235161336
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/QueryPlanningTracker.scala
---
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23096
cc @hvanhovell @gatorsmile
This is different from the existing metrics for rules as it is query
specific. We might want to replace that one with this in the future
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/23096
[SPARK-26129][SQL] Instrumentation for query planning time
## What changes were proposed in this pull request?
We currently don't have good visibility into query planning time (analysis
vs
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/23054#discussion_r234569150
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1594,6 +1594,15 @@ object SQLConf {
"WHERE, which
1 - 100 of 19261 matches
Mail list logo
