[spark] branch master updated: [SPARK-34355][CORE][SQL][FOLLOWUP] Log commit time in all File Writer

2021-02-08 Thread kabhwan 
This is an automated email from the ASF dual-hosted git repository.

kabhwan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new 7ea3a33  [SPARK-34355][CORE][SQL][FOLLOWUP] Log commit time in all 
File Writer
7ea3a33 is described below

commit 7ea3a336b99915f09174a4c3e47fa17f30b88890
Author: Angerszh 
AuthorDate: Tue Feb 9 16:05:39 2021 +0900

[SPARK-34355][CORE][SQL][FOLLOWUP] Log commit time in all File Writer

### What changes were proposed in this pull request?
When doing https://issues.apache.org/jira/browse/SPARK-34399 based  on 
https://github.com/apache/spark/pull/31471
Found FileBatchWrite will use `FileFormatWrite.processStates()` too. We 
need log commit duration  in other writer too.
In this pr:

1. Extract a commit job method in SparkHadoopWriter
2. address other commit writer

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
No

Closes #31520 from AngersZh/SPARK-34355-followup.

Authored-by: Angerszh 
Signed-off-by: Jungtaek Lim (HeartSaVioR) 
---
 .../main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala | 5 +++--
 .../apache/spark/sql/execution/datasources/v2/FileBatchWrite.scala  | 6 --
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git 
a/core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala 
b/core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala
index 37b4708..4eeec63 100644
--- a/core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala
+++ b/core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala
@@ -96,8 +96,9 @@ object SparkHadoopWriter extends Logging {
   iterator = iter)
   })
 
-  committer.commitJob(jobContext, ret)
-  logInfo(s"Job ${jobContext.getJobID} committed.")
+  logInfo(s"Start to commit write Job ${jobContext.getJobID}.")
+  val (_, duration) = Utils.timeTakenMs { committer.commitJob(jobContext, 
ret) }
+  logInfo(s"Write Job ${jobContext.getJobID} committed. Elapsed time: 
$duration ms.")
 } catch {
   case cause: Throwable =>
 logError(s"Aborting job ${jobContext.getJobID}.", cause)
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileBatchWrite.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileBatchWrite.scala
index 266c834..7227e48 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileBatchWrite.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileBatchWrite.scala
@@ -23,6 +23,7 @@ import org.apache.spark.internal.io.FileCommitProtocol
 import org.apache.spark.sql.connector.write.{BatchWrite, DataWriterFactory, 
PhysicalWriteInfo, WriterCommitMessage}
 import org.apache.spark.sql.execution.datasources.{WriteJobDescription, 
WriteTaskResult}
 import org.apache.spark.sql.execution.datasources.FileFormatWriter.processStats
+import org.apache.spark.util.Utils
 
 class FileBatchWrite(
 job: Job,
@@ -31,8 +32,9 @@ class FileBatchWrite(
   extends BatchWrite with Logging {
   override def commit(messages: Array[WriterCommitMessage]): Unit = {
 val results = messages.map(_.asInstanceOf[WriteTaskResult])
-committer.commitJob(job, results.map(_.commitMsg))
-logInfo(s"Write Job ${description.uuid} committed.")
+logInfo(s"Start to commit write Job ${description.uuid}.")
+val (_, duration) = Utils.timeTakenMs { committer.commitJob(job, 
results.map(_.commitMsg)) }
+logInfo(s"Write Job ${description.uuid} committed. Elapsed time: $duration 
ms.")
 
 processStats(description.statsTrackers, results.map(_.summary.stats))
 logInfo(s"Finished processing stats for write job ${description.uuid}.")


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-2.4 updated: [SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop should clean up K8s resources

2021-02-08 Thread dongjoon 
This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-2.4 by this push:
 new fa78e68  [SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop 
should clean up K8s resources
fa78e68 is described below

commit fa78e68c2f28c5bac056ad0402cba110b3faf50c
Author: Dongjoon Hyun 
AuthorDate: Mon Feb 8 21:47:23 2021 -0800

[SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop should clean up 
K8s resources

This PR aims to fix `KubernetesClusterSchedulerBackend.stop` to wrap 
`super.stop` with `Utils.tryLogNonFatalError`.


[CoarseGrainedSchedulerBackend.stop](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L559)
 may throw `SparkException` and this causes K8s resource (pod and configmap) 
leakage.

No. This is a bug fix.

Pass the CI with the newly added test case.

Closes #31533 from dongjoon-hyun/SPARK-34407.

Authored-by: Dongjoon Hyun 
Signed-off-by: Dongjoon Hyun 
(cherry picked from commit ea339c38b43c59931257386efdd490507f7de64d)
Signed-off-by: Dongjoon Hyun 
---
 .../cluster/k8s/KubernetesClusterSchedulerBackend.scala   |  6 +-
 .../cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala  | 11 ++-
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git 
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
 
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
index bdd4134..bc89002 100644
--- 
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
+++ 
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
@@ -81,7 +81,11 @@ private[spark] class KubernetesClusterSchedulerBackend(
   }
 
   override def stop(): Unit = {
-super.stop()
+// When `CoarseGrainedSchedulerBackend.stop` throws `SparkException`,
+// K8s cluster scheduler should log and proceed in order to delete the K8s 
cluster resources.
+Utils.tryLogNonFatalError {
+  super.stop()
+}
 
 Utils.tryLogNonFatalError {
   snapshotsStore.stop()
diff --git 
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
 
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
index fbff1d7..90dfc0c 100644
--- 
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
+++ 
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
@@ -28,7 +28,7 @@ import org.apache.spark.deploy.k8s.Constants._
 import org.apache.spark.deploy.k8s.Fabric8Aliases._
 import org.apache.spark.rpc.{RpcEndpoint, RpcEndpointRef, RpcEnv}
 import org.apache.spark.scheduler.{ExecutorKilled, TaskSchedulerImpl}
-import 
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
+import 
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.{RemoveExecutor,
 StopDriver}
 import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 import 
org.apache.spark.scheduler.cluster.k8s.ExecutorLifecycleTestUtils.TEST_SPARK_APP_ID
 
@@ -147,4 +147,13 @@ class KubernetesClusterSchedulerBackendSuite extends 
SparkFunSuite with BeforeAn
 verify(podAllocator).setTotalExpectedExecutors(5)
   }
 
+  test("SPARK-34407: CoarseGrainedSchedulerBackend.stop may throw 
SparkException") {
+schedulerBackendUnderTest.start()
+
+when(driverEndpointRef.askSync[Boolean](StopDriver)).thenThrow(new 
RuntimeException)
+schedulerBackendUnderTest.stop()
+
+// Verify the last operation of `schedulerBackendUnderTest.stop`.
+verify(kubernetesClient).close()
+  }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.0 updated: [SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop should clean up K8s resources

2021-02-08 Thread dongjoon 
This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new b560584  [SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop 
should clean up K8s resources
b560584 is described below

commit b56058456fa10695ee08a17f6cf7080a5edc96e7
Author: Dongjoon Hyun 
AuthorDate: Mon Feb 8 21:47:23 2021 -0800

[SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop should clean up 
K8s resources

This PR aims to fix `KubernetesClusterSchedulerBackend.stop` to wrap 
`super.stop` with `Utils.tryLogNonFatalError`.


[CoarseGrainedSchedulerBackend.stop](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L559)
 may throw `SparkException` and this causes K8s resource (pod and configmap) 
leakage.

No. This is a bug fix.

Pass the CI with the newly added test case.

Closes #31533 from dongjoon-hyun/SPARK-34407.

Authored-by: Dongjoon Hyun 
Signed-off-by: Dongjoon Hyun 
(cherry picked from commit ea339c38b43c59931257386efdd490507f7de64d)
Signed-off-by: Dongjoon Hyun 
---
 .../cluster/k8s/KubernetesClusterSchedulerBackend.scala  |  6 +-
 .../cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala | 12 +++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git 
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
 
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
index 105841a..f801802 100644
--- 
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
+++ 
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
@@ -83,7 +83,11 @@ private[spark] class KubernetesClusterSchedulerBackend(
   }
 
   override def stop(): Unit = {
-super.stop()
+// When `CoarseGrainedSchedulerBackend.stop` throws `SparkException`,
+// K8s cluster scheduler should log and proceed in order to delete the K8s 
cluster resources.
+Utils.tryLogNonFatalError {
+  super.stop()
+}
 
 Utils.tryLogNonFatalError {
   snapshotsStore.stop()
diff --git 
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
 
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
index 7e1e39c..260be03 100644
--- 
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
+++ 
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
@@ -33,7 +33,7 @@ import org.apache.spark.deploy.k8s.Constants._
 import org.apache.spark.deploy.k8s.Fabric8Aliases._
 import org.apache.spark.rpc.{RpcEndpoint, RpcEndpointRef, RpcEnv}
 import org.apache.spark.scheduler.{ExecutorKilled, TaskSchedulerImpl}
-import 
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
+import 
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.{RemoveExecutor,
 StopDriver}
 import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 import 
org.apache.spark.scheduler.cluster.k8s.ExecutorLifecycleTestUtils.TEST_SPARK_APP_ID
 
@@ -170,4 +170,14 @@ class KubernetesClusterSchedulerBackendSuite extends 
SparkFunSuite with BeforeAn
   TimeUnit.MILLISECONDS)
 verify(labeledPods).delete()
   }
+
+  test("SPARK-34407: CoarseGrainedSchedulerBackend.stop may throw 
SparkException") {
+schedulerBackendUnderTest.start()
+
+when(driverEndpointRef.askSync[Boolean](StopDriver)).thenThrow(new 
RuntimeException)
+schedulerBackendUnderTest.stop()
+
+// Verify the last operation of `schedulerBackendUnderTest.stop`.
+verify(kubernetesClient).close()
+  }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.1 updated: [SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop should clean up K8s resources

2021-02-08 Thread dongjoon 
This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new b0ffbf7  [SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop 
should clean up K8s resources
b0ffbf7 is described below

commit b0ffbf7e8889ee0d3e086c997351bf67876c9a29
Author: Dongjoon Hyun 
AuthorDate: Mon Feb 8 21:47:23 2021 -0800

[SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop should clean up 
K8s resources

### What changes were proposed in this pull request?

This PR aims to fix `KubernetesClusterSchedulerBackend.stop` to wrap 
`super.stop` with `Utils.tryLogNonFatalError`.

### Why are the changes needed?


[CoarseGrainedSchedulerBackend.stop](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L559)
 may throw `SparkException` and this causes K8s resource (pod and configmap) 
leakage.

### Does this PR introduce _any_ user-facing change?

No. This is a bug fix.

### How was this patch tested?

Pass the CI with the newly added test case.

Closes #31533 from dongjoon-hyun/SPARK-34407.

Authored-by: Dongjoon Hyun 
Signed-off-by: Dongjoon Hyun 
(cherry picked from commit ea339c38b43c59931257386efdd490507f7de64d)
Signed-off-by: Dongjoon Hyun 
---
 .../cluster/k8s/KubernetesClusterSchedulerBackend.scala  |  6 +-
 .../cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala | 12 +++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git 
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
 
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
index 30a7c65..78862bc 100644
--- 
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
+++ 
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
@@ -99,7 +99,11 @@ private[spark] class KubernetesClusterSchedulerBackend(
   }
 
   override def stop(): Unit = {
-super.stop()
+// When `CoarseGrainedSchedulerBackend.stop` throws `SparkException`,
+// K8s cluster scheduler should log and proceed in order to delete the K8s 
cluster resources.
+Utils.tryLogNonFatalError {
+  super.stop()
+}
 
 Utils.tryLogNonFatalError {
   snapshotsStore.stop()
diff --git 
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
 
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
index 632c564..b0dd40d 100644
--- 
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
+++ 
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
@@ -34,7 +34,7 @@ import org.apache.spark.deploy.k8s.Fabric8Aliases._
 import org.apache.spark.resource.{ResourceProfile, ResourceProfileManager}
 import org.apache.spark.rpc.{RpcEndpoint, RpcEndpointRef, RpcEnv}
 import org.apache.spark.scheduler.{ExecutorKilled, LiveListenerBus, 
TaskSchedulerImpl}
-import 
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
+import 
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.{RemoveExecutor,
 StopDriver}
 import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 import 
org.apache.spark.scheduler.cluster.k8s.ExecutorLifecycleTestUtils.TEST_SPARK_APP_ID
 
@@ -189,4 +189,14 @@ class KubernetesClusterSchedulerBackendSuite extends 
SparkFunSuite with BeforeAn
   TimeUnit.MILLISECONDS)
 verify(labeledPods).delete()
   }
+
+  test("SPARK-34407: CoarseGrainedSchedulerBackend.stop may throw 
SparkException") {
+schedulerBackendUnderTest.start()
+
+when(driverEndpointRef.askSync[Boolean](StopDriver)).thenThrow(new 
RuntimeException)
+schedulerBackendUnderTest.stop()
+
+// Verify the last operation of `schedulerBackendUnderTest.stop`.
+verify(kubernetesClient).close()
+  }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated: [SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop should clean up K8s resources

2021-02-08 Thread dongjoon 
This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new ea339c3  [SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop 
should clean up K8s resources
ea339c3 is described below

commit ea339c38b43c59931257386efdd490507f7de64d
Author: Dongjoon Hyun 
AuthorDate: Mon Feb 8 21:47:23 2021 -0800

[SPARK-34407][K8S] KubernetesClusterSchedulerBackend.stop should clean up 
K8s resources

### What changes were proposed in this pull request?

This PR aims to fix `KubernetesClusterSchedulerBackend.stop` to wrap 
`super.stop` with `Utils.tryLogNonFatalError`.

### Why are the changes needed?


[CoarseGrainedSchedulerBackend.stop](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L559)
 may throw `SparkException` and this causes K8s resource (pod and configmap) 
leakage.

### Does this PR introduce _any_ user-facing change?

No. This is a bug fix.

### How was this patch tested?

Pass the CI with the newly added test case.

Closes #31533 from dongjoon-hyun/SPARK-34407.

Authored-by: Dongjoon Hyun 
Signed-off-by: Dongjoon Hyun 
---
 .../cluster/k8s/KubernetesClusterSchedulerBackend.scala  |  6 +-
 .../cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala | 12 +++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git 
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
 
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
index cb3378c..cd3b36c 100644
--- 
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
+++ 
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
@@ -101,7 +101,11 @@ private[spark] class KubernetesClusterSchedulerBackend(
   }
 
   override def stop(): Unit = {
-super.stop()
+// When `CoarseGrainedSchedulerBackend.stop` throws `SparkException`,
+// K8s cluster scheduler should log and proceed in order to delete the K8s 
cluster resources.
+Utils.tryLogNonFatalError {
+  super.stop()
+}
 
 Utils.tryLogNonFatalError {
   snapshotsStore.stop()
diff --git 
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
 
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
index fdc3633..bbae9dd 100644
--- 
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
+++ 
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
@@ -34,7 +34,7 @@ import org.apache.spark.deploy.k8s.Fabric8Aliases._
 import org.apache.spark.resource.{ResourceProfile, ResourceProfileManager}
 import org.apache.spark.rpc.{RpcEndpoint, RpcEndpointRef, RpcEnv}
 import org.apache.spark.scheduler.{ExecutorKilled, LiveListenerBus, 
TaskSchedulerImpl}
-import 
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
+import 
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.{RemoveExecutor,
 StopDriver}
 import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 import 
org.apache.spark.scheduler.cluster.k8s.ExecutorLifecycleTestUtils.TEST_SPARK_APP_ID
 
@@ -189,4 +189,14 @@ class KubernetesClusterSchedulerBackendSuite extends 
SparkFunSuite with BeforeAn
   TimeUnit.MILLISECONDS)
 verify(labeledPods).delete()
   }
+
+  test("SPARK-34407: CoarseGrainedSchedulerBackend.stop may throw 
SparkException") {
+schedulerBackendUnderTest.start()
+
+when(driverEndpointRef.askSync[Boolean](StopDriver)).thenThrow(new 
RuntimeException)
+schedulerBackendUnderTest.stop()
+
+// Verify the last operation of `schedulerBackendUnderTest.stop`.
+verify(kubernetesClient).close()
+  }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.0 updated: [SPARK-34405][CORE] Fix mean value of timersLabels in the PrometheusServlet class

2021-02-08 Thread dongjoon 
This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new 83f3c2e  [SPARK-34405][CORE] Fix mean value of timersLabels in the 
PrometheusServlet class
83f3c2e is described below

commit 83f3c2ed25a4a773678fd317ab5bd01f4bfb5427
Author: wyp 
AuthorDate: Mon Feb 8 21:18:29 2021 -0800

[SPARK-34405][CORE] Fix mean value of timersLabels in the PrometheusServlet 
class

### What changes were proposed in this pull request?
The getMetricsSnapshot method of the PrometheusServlet class has a wrong 
value, It should be taking the mean value but it's taking the max value.

### Why are the changes needed?

The mean value of timersLabels in the PrometheusServlet class is wrong, You 
can look at line 105 of this class: L105.

```
sb.append(s"${prefix}Mean$timersLabels ${snapshot.getMax}\n")
```
it should be
```
sb.append(s"${prefix}Mean$timersLabels ${snapshot.getMean}\n")
```

### Does this PR introduce _any_ user-facing change?

No
### How was this patch tested?


![image](https://user-images.githubusercontent.com/5170878/107313576-cc199280-6acd-11eb-9384-b6abf71c0f90.png)

Closes #31532 from 397090770/SPARK-34405.

Authored-by: wyp 
Signed-off-by: Dongjoon Hyun 
(cherry picked from commit a1e75edc39c11e85d8a4917c3e82282fa974be96)
Signed-off-by: Dongjoon Hyun 
---
 .../main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala 
b/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala
index 59b863b..ae10997 100644
--- a/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala
+++ b/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala
@@ -102,7 +102,7 @@ private[spark] class PrometheusServlet(
   val snapshot = timer.getSnapshot
   sb.append(s"${prefix}Count$timersLabels ${timer.getCount}\n")
   sb.append(s"${prefix}Max$timersLabels ${snapshot.getMax}\n")
-  sb.append(s"${prefix}Mean$timersLabels ${snapshot.getMax}\n")
+  sb.append(s"${prefix}Mean$timersLabels ${snapshot.getMean}\n")
   sb.append(s"${prefix}Min$timersLabels ${snapshot.getMin}\n")
   sb.append(s"${prefix}50thPercentile$timersLabels 
${snapshot.getMedian}\n")
   sb.append(s"${prefix}75thPercentile$timersLabels 
${snapshot.get75thPercentile}\n")


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.1 updated: [SPARK-34405][CORE] Fix mean value of timersLabels in the PrometheusServlet class

2021-02-08 Thread dongjoon 
This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 7047923  [SPARK-34405][CORE] Fix mean value of timersLabels in the 
PrometheusServlet class
7047923 is described below

commit 7047923abbb392a3573c0269ab3f9233c39865d2
Author: wyp 
AuthorDate: Mon Feb 8 21:18:29 2021 -0800

[SPARK-34405][CORE] Fix mean value of timersLabels in the PrometheusServlet 
class

### What changes were proposed in this pull request?
The getMetricsSnapshot method of the PrometheusServlet class has a wrong 
value, It should be taking the mean value but it's taking the max value.

### Why are the changes needed?

The mean value of timersLabels in the PrometheusServlet class is wrong, You 
can look at line 105 of this class: L105.

```
sb.append(s"${prefix}Mean$timersLabels ${snapshot.getMax}\n")
```
it should be
```
sb.append(s"${prefix}Mean$timersLabels ${snapshot.getMean}\n")
```

### Does this PR introduce _any_ user-facing change?

No
### How was this patch tested?


![image](https://user-images.githubusercontent.com/5170878/107313576-cc199280-6acd-11eb-9384-b6abf71c0f90.png)

Closes #31532 from 397090770/SPARK-34405.

Authored-by: wyp 
Signed-off-by: Dongjoon Hyun 
(cherry picked from commit a1e75edc39c11e85d8a4917c3e82282fa974be96)
Signed-off-by: Dongjoon Hyun 
---
 .../main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala 
b/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala
index e9c2974..0f8fbd3 100644
--- a/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala
+++ b/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala
@@ -102,7 +102,7 @@ private[spark] class PrometheusServlet(
   val snapshot = timer.getSnapshot
   sb.append(s"${prefix}Count$timersLabels ${timer.getCount}\n")
   sb.append(s"${prefix}Max$timersLabels ${snapshot.getMax}\n")
-  sb.append(s"${prefix}Mean$timersLabels ${snapshot.getMax}\n")
+  sb.append(s"${prefix}Mean$timersLabels ${snapshot.getMean}\n")
   sb.append(s"${prefix}Min$timersLabels ${snapshot.getMin}\n")
   sb.append(s"${prefix}50thPercentile$timersLabels 
${snapshot.getMedian}\n")
   sb.append(s"${prefix}75thPercentile$timersLabels 
${snapshot.get75thPercentile}\n")


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (37fe8c6 -> a1e75ed)

2021-02-08 Thread dongjoon 
This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 37fe8c6  [SPARK-34395][SQL] Clean up unused code for code 
simplifications
 add a1e75ed  [SPARK-34405][CORE] Fix mean value of timersLabels in the 
PrometheusServlet class

No new revisions were added by this update.

Summary of changes:
 .../main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.1 updated: Revert "[SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on windows system"

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new c7e9da9  Revert "[SPARK-34352][SQL] Improve SQLQueryTestSuite so as 
could run on windows system"
c7e9da9 is described below

commit c7e9da93c6c47294a917dcd5e4f6b9ea89ae12a4
Author: HyukjinKwon 
AuthorDate: Tue Feb 9 12:03:11 2021 +0900

Revert "[SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on 
windows system"

This reverts commit db8db0da1c2da24c191b0b89a0fcaa55eafeb7ef.
---
 .../src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala   | 8 ++--
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
index b573f94..02c6fba 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
@@ -154,14 +154,10 @@ class SQLQueryTestSuite extends QueryTest with 
SharedSparkSession with SQLHelper
 // Fewer shuffle partitions to speed up testing.
 .set(SQLConf.SHUFFLE_PARTITIONS, 4)
 
-  // SPARK-32106 Since we add SQL test 'transform.sql' will use `cat` command,
-  // here we need to ignore it.
-  private val otherIgnoreList =
-if (TestUtils.testCommandAvailable("/bin/bash")) Nil else 
Set("transform.sql")
   /** List of test cases to ignore, in lower cases. */
   protected def ignoreList: Set[String] = Set(
-"ignored.sql" // Do NOT remove this one. It is here to test the ignore 
functionality.
-  ) ++ otherIgnoreList
+"ignored.sql"   // Do NOT remove this one. It is here to test the ignore 
functionality.
+  )
 
   // Create all the test cases.
   listTestCases.foreach(createScalaTestCase)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (e65b28c -> 37fe8c6)

2021-02-08 Thread srowen 
This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from e65b28c  [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run 
on windows system
 add 37fe8c6  [SPARK-34395][SQL] Clean up unused code for code 
simplifications

No new revisions were added by this update.

Summary of changes:
 .../sql/catalyst/expressions/StringExpressionsSuite.scala  | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (777d51e -> e65b28c)

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 777d51e  [SPARK-34374][SQL][DSTREAM] Use standard methods to extract 
keys or values from a Map
 add e65b28c  [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run 
on windows system

No new revisions were added by this update.

Summary of changes:
 .../test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala   | 11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.1 updated: [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on windows system

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new db8db0d  [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run 
on windows system
db8db0d is described below

commit db8db0da1c2da24c191b0b89a0fcaa55eafeb7ef
Author: gengjiaan 
AuthorDate: Tue Feb 9 10:58:58 2021 +0900

[SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on windows 
system

### What changes were proposed in this pull request?
The current implement of `SQLQueryTestSuite` cannot run on windows system.
Becasue the code below will fail on windows system:
`assume(TestUtils.testCommandAvailable("/bin/bash"))`

For operation system that cannot support `/bin/bash`, we just skip some 
tests.

### Why are the changes needed?
SQLQueryTestSuite has a bug on windows system.

### Does this PR introduce _any_ user-facing change?
'No'.

### How was this patch tested?
Jenkins test

Closes #31466 from beliefer/SPARK-34352.

Authored-by: gengjiaan 
Signed-off-by: HyukjinKwon 
(cherry picked from commit e65b28cf7d9680ebdf96833a6f2d38ffd61c7d21)
Signed-off-by: HyukjinKwon 
---
 .../src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala   | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
index 02c6fba..b573f94 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
@@ -154,10 +154,14 @@ class SQLQueryTestSuite extends QueryTest with 
SharedSparkSession with SQLHelper
 // Fewer shuffle partitions to speed up testing.
 .set(SQLConf.SHUFFLE_PARTITIONS, 4)
 
+  // SPARK-32106 Since we add SQL test 'transform.sql' will use `cat` command,
+  // here we need to ignore it.
+  private val otherIgnoreList =
+if (TestUtils.testCommandAvailable("/bin/bash")) Nil else 
Set("transform.sql")
   /** List of test cases to ignore, in lower cases. */
   protected def ignoreList: Set[String] = Set(
-"ignored.sql"   // Do NOT remove this one. It is here to test the ignore 
functionality.
-  )
+"ignored.sql" // Do NOT remove this one. It is here to test the ignore 
functionality.
+  ) ++ otherIgnoreList
 
   // Create all the test cases.
   listTestCases.foreach(createScalaTestCase)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (3b26bc2 -> 777d51e)

2021-02-08 Thread srowen 
This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 3b26bc2  [SPARK-34168][SQL] Support DPP in AQE when the join is 
Broadcast hash join at the beginning
 add 777d51e  [SPARK-34374][SQL][DSTREAM] Use standard methods to extract 
keys or values from a Map

No new revisions were added by this update.

Summary of changes:
 .../main/scala/org/apache/spark/sql/execution/datasources/rules.scala  | 3 +--
 .../apache/spark/streaming/scheduler/ReceiverSchedulingPolicy.scala| 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (c92e408 -> 3b26bc2)

2021-02-08 Thread wenchen 
This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from c92e408  [SPARK-34388][SQL] Propagate the registered UDF name to 
ScalaUDF, ScalaUDAF and ScalaAggregator
 add 3b26bc2  [SPARK-34168][SQL] Support DPP in AQE when the join is 
Broadcast hash join at the beginning

No new revisions were added by this update.

Summary of changes:
 ...c.scala => SubqueryAdaptiveBroadcastExec.scala} | 37 ++---
 .../execution/adaptive/AdaptiveSparkPlanExec.scala |  5 +-
 .../adaptive/CustomShuffleReaderExec.scala |  4 +-
 .../adaptive/InsertAdaptiveSparkPlan.scala | 18 +--
 .../PlanAdaptiveDynamicPruningFilters.scala| 59 +
 .../adaptive/PlanAdaptiveSubqueries.scala  |  9 ++--
 .../sql/execution/adaptive/QueryStageExec.scala| 27 --
 .../spark/sql/DynamicPartitionPruningSuite.scala   | 61 --
 .../execution/CoalesceShufflePartitionsSuite.scala |  4 +-
 .../execution/RemoveRedundantProjectsSuite.scala   |  3 ++
 .../adaptive/AdaptiveQueryExecSuite.scala  |  4 +-
 11 files changed, 173 insertions(+), 58 deletions(-)
 copy 
sql/core/src/main/scala/org/apache/spark/sql/execution/{datasources/v2/AlterTableExec.scala
 => SubqueryAdaptiveBroadcastExec.scala} (52%)
 create mode 100644 
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

svn commit: r45893 - in /dev/spark/v3.1.1-rc2-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _site/api/java/org/apache/parqu

2021-02-08 Thread gurwls223 
Author: gurwls223
Date: Mon Feb  8 15:19:40 2021
New Revision: 45893

Log:
Apache Spark v3.1.1-rc2 docs


[This commit notification would consist of 2258 parts, 
which exceeds the limit of 50 ones, so it was shortened to the summary.]

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

svn commit: r45892 - /dev/spark/v3.1.1-rc2-bin/

2021-02-08 Thread gurwls223 
Author: gurwls223
Date: Mon Feb  8 14:57:31 2021
New Revision: 45892

Log:
Apache Spark v3.1.1-rc2

Added:
dev/spark/v3.1.1-rc2-bin/
dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz   (with props)
dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz.asc
dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz.sha512
dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz   (with props)
dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz.asc
dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz.sha512
dev/spark/v3.1.1-rc2-bin/spark-3.1.1-bin-hadoop2.7.tgz   (with props)
dev/spark/v3.1.1-rc2-bin/spark-3.1.1-bin-hadoop2.7.tgz.asc
dev/spark/v3.1.1-rc2-bin/spark-3.1.1-bin-hadoop2.7.tgz.sha512
dev/spark/v3.1.1-rc2-bin/spark-3.1.1-bin-hadoop3.2.tgz   (with props)
dev/spark/v3.1.1-rc2-bin/spark-3.1.1-bin-hadoop3.2.tgz.asc
dev/spark/v3.1.1-rc2-bin/spark-3.1.1-bin-hadoop3.2.tgz.sha512
dev/spark/v3.1.1-rc2-bin/spark-3.1.1-bin-without-hadoop.tgz   (with props)
dev/spark/v3.1.1-rc2-bin/spark-3.1.1-bin-without-hadoop.tgz.asc
dev/spark/v3.1.1-rc2-bin/spark-3.1.1-bin-without-hadoop.tgz.sha512
dev/spark/v3.1.1-rc2-bin/spark-3.1.1.tgz   (with props)
dev/spark/v3.1.1-rc2-bin/spark-3.1.1.tgz.asc
dev/spark/v3.1.1-rc2-bin/spark-3.1.1.tgz.sha512

Added: dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz
==
Binary file - no diff available.

Propchange: dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz.asc
==
--- dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz.asc (added)
+++ dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz.asc Mon Feb  8 14:57:31 2021
@@ -0,0 +1,14 @@
+-BEGIN PGP SIGNATURE-
+
+iQHJBAABCgAzFiEEPS9pzu1eMPYlKTWQP/fE6DTZ6kQFAmAhSTUVHGd1cndsczIy
+M0BhcGFjaGUub3JnAAoJED/3xOg02epEmx4L/3yItCFI2ZDLqMpCqnpY5gPAlqkv
+g4tvEh8uajL1X5CmAj1lYIeHg7AWMMca2zgQqdUxa47cn+lmuH73lMFpOnIKsDxl
+zB2rjmf5jRbP1DvfrZx905yOPxjIqGOwv9z4GBgWUAbrXn/7KIOXrABNz2YtxDtH
+HQXw7okEj0V+9r2E1VohBDPSseGXFLuOEA0bU9dwwDCSzQEdDSvw5HQNYMq/GIbK
+nrEcfSWj2wg3JvwhzQY8Sj1M8s8cpeNOtGvbwx+ePRRTsZw9VyNXy/gGuBC/TupA
+l4BbdmNKcUc6pVaRuh6Lnz4XWIqFsdo5Cli4nVmV46qABZ64DeO1geu/N65GfDFT
+BfpsjeF7aYWmtgs7Q/r2VxivdHr9yBbiKyJr3b/uXbe2xQmh0AuRy4BXGa+YqBCo
+29gTp7DBlcH0wh7cCowZIMl9B8o4naSjxFDP5U+QLXm1qSAMlI3zC/OF7ppIhEsK
+rF4dCKQP8TeWhGlkYsTaLzze8mMWNiKeyd8eBw==
+=sjUE
+-END PGP SIGNATURE-

Added: dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz.sha512
==
--- dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz.sha512 (added)
+++ dev/spark/v3.1.1-rc2-bin/SparkR_3.1.1.tar.gz.sha512 Mon Feb  8 14:57:31 2021
@@ -0,0 +1,3 @@
+SparkR_3.1.1.tar.gz: 6AC592AC FE8D3D43 9B11CD4A E4981B55 26719355 78226ABF
+ C767D1D8 3DA623C3 7C3945F0 45BB4EB4 EFFB6304 041C3229
+ A23DB2E0 AE880748 3582F67C A2541807

Added: dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz
==
Binary file - no diff available.

Propchange: dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz.asc
==
--- dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz.asc (added)
+++ dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz.asc Mon Feb  8 14:57:31 2021
@@ -0,0 +1,14 @@
+-BEGIN PGP SIGNATURE-
+
+iQHJBAABCgAzFiEEPS9pzu1eMPYlKTWQP/fE6DTZ6kQFAmAhSTcVHGd1cndsczIy
+M0BhcGFjaGUub3JnAAoJED/3xOg02epED3QMALk7CxLUvSZijhqBcPQ0m/2SKp6m
+H84kqNpgvEzayl3nXW2AwhBkqwXzKSuMmEMo1u+RJhv3OBPcEgEIlCtp6RUxG8ga
++E1g/YZC8dmgkgTavA3EvyLtxC5/rST7eSqBojhOKIfG8f5IuuCAFulIkM9DFJ2q
+2XE9hTAmD6iIOBQpu8nM2KIhvLRXDSwEWvrxUAgtEapRVQtHLUV8BXS7+arxFVls
+ZQUkuSoOfL9SK1tCpxqQfpyk0YkWHs6ETnqo2Qv6fCZp9PCNyia7L03zQ10Q+EEk
+sis38xygGDthunAJLNMwwrs6qGOC0WFAZpPH4IZ1vaK4YCBqrLX54nHaKGyM0B9E
+Yq0w6oVtcrxlLFk347O+Od6L2rPZih/DQ1hRDP1tt4z95u1Yf+w+gs7Pc2D+3qeF
+KAfPALdloopc6QFdZTgvTG6N5MT5Q7bdkl986oYQeRd2b9eEI7oA7Xzc1W4TRBgf
+klGMxzm0hJcPjDEiyIgwykqjvoMP6DhSuuSZ1w==
+=h8AG
+-END PGP SIGNATURE-

Added: dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz.sha512
==
--- dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz.sha512 (added)
+++ dev/spark/v3.1.1-rc2-bin/pyspark-3.1.1.tar.gz.sha512 Mon Feb  8 14:57:31 
2021
@@ -0,0 +1,3 @@
+pyspark-3.1.1.tar.gz: 5EA08A9C 3DA610E6 39564781 354460DF 5A5D881C 951420FC
+  5AB18B51 BBB2C3A4 6CE4E33F 3BB4EE5F 3C2C52AD 50B03DA9
+  86E66F31 09C87169 3BB348E2 1CC50B83

Added:

[spark] branch master updated: [MINOR][SQL][FOLLOW-UP] Add assertion to FixedLengthRowBasedKeyValueBatch

2021-02-08 Thread srowen 
This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new d1131bc  [MINOR][SQL][FOLLOW-UP] Add assertion to 
FixedLengthRowBasedKeyValueBatch
d1131bc is described below

commit d1131bc85028ea0f78ac9ef73bba731080f1ff6a
Author: yliou 
AuthorDate: Mon Feb 8 08:46:01 2021 -0600

[MINOR][SQL][FOLLOW-UP] Add assertion to FixedLengthRowBasedKeyValueBatch

### What changes were proposed in this pull request?
Adds an assert to `FixedLengthRowBasedKeyValueBatch#appendRow` method to 
check the incoming vlen and klen by comparing them with the lengths stored as 
member variables as followup to https://github.com/apache/spark/pull/30788

### Why are the changes needed?
Add assert statement to catch similar bugs in future.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Ran some tests locally, though not easy to test.

Closes #31447 from yliou/SPARK-33726-Assert.

Authored-by: yliou 
Signed-off-by: Sean Owen 
---
 .../sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java  | 2 ++
 1 file changed, 2 insertions(+)

diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java
index df52f9c..25400be 100644
--- 
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java
@@ -46,6 +46,8 @@ public final class FixedLengthRowBasedKeyValueBatch extends 
RowBasedKeyValueBatc
   public UnsafeRow appendRow(Object kbase, long koff, int klen,
  Object vbase, long voff, int vlen) {
 // if run out of max supported rows or page size, return null
+assert(vlen == this.vlen);
+assert(klen == this.klen);
 if (numRows >= capacity || page == null || page.size() - pageCursor < 
recordLength) {
   return null;
 }


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.0 updated: [SPARK-33438][SQL] Eagerly init objects with defined SQL Confs for command `set -v`

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new f611788  [SPARK-33438][SQL] Eagerly init objects with defined SQL 
Confs for command `set -v`
f611788 is described below

commit f611788e5a18e4bf56df7d3e3b0263384ec10bff
Author: Linhong Liu 
AuthorDate: Mon Feb 8 22:48:28 2021 +0900

[SPARK-33438][SQL] Eagerly init objects with defined SQL Confs for command 
`set -v`

### What changes were proposed in this pull request?
In Spark, `set -v` is defined as "Queries all properties that are defined 
in the SQLConf of the sparkSession".
But there are other external modules that also define properties and 
register them to SQLConf. In this case,
it can't be displayed by `set -v` until the conf object is initiated (i.e. 
calling the object at least once).

In this PR, I propose to eagerly initiate all the objects registered to 
SQLConf, so that `set -v` will always output
the completed properties.

### Why are the changes needed?
Improve the `set -v` command to produces completed and  deterministic 
results

### Does this PR introduce _any_ user-facing change?
`set -v` command will dump more configs

### How was this patch tested?
existing tests

Closes #30363 from linhongliu-db/set-v.

Authored-by: Linhong Liu 
Signed-off-by: HyukjinKwon 
(cherry picked from commit 037bfb2dbcb73cfbd73f0fd9abe0b38789a182a2)
Signed-off-by: HyukjinKwon 
---
 .../org/apache/spark/sql/internal/SQLConf.scala| 24 ++
 .../spark/sql/api/python/PythonSQLUtils.scala  | 15 +-
 2 files changed, 25 insertions(+), 14 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index f55546f..a2aa6f8 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -25,6 +25,7 @@ import java.util.zip.Deflater
 import scala.collection.JavaConverters._
 import scala.collection.immutable
 import scala.util.Try
+import scala.util.control.NonFatal
 import scala.util.matching.Regex
 
 import org.apache.hadoop.fs.Path
@@ -35,6 +36,7 @@ import org.apache.spark.internal.config._
 import org.apache.spark.internal.config.{IGNORE_MISSING_FILES => 
SPARK_IGNORE_MISSING_FILES}
 import org.apache.spark.network.util.ByteUnit
 import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.ScalaReflection
 import org.apache.spark.sql.catalyst.analysis.{HintErrorLogger, Resolver}
 import org.apache.spark.sql.catalyst.expressions.CodegenObjectFactoryMode
 import org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator
@@ -3292,6 +3294,27 @@ class SQLConf extends Serializable with Logging {
 }
   }
 
+  private var definedConfsLoaded = false
+  /**
+   * Init [[StaticSQLConf]] and [[org.apache.spark.sql.hive.HiveUtils]] so 
that all the defined
+   * SQL Configurations will be registered to SQLConf
+   */
+  private def loadDefinedConfs(): Unit = {
+if (!definedConfsLoaded) {
+  definedConfsLoaded = true
+  // Force to register static SQL configurations
+  StaticSQLConf
+  try {
+// Force to register SQL configurations from Hive module
+val symbol = 
ScalaReflection.mirror.staticModule("org.apache.spark.sql.hive.HiveUtils")
+ScalaReflection.mirror.reflectModule(symbol).instance
+  } catch {
+case NonFatal(e) =>
+  logWarning("SQL configurations from Hive module is not loaded", e)
+  }
+}
+  }
+
   /**
* Return all the configuration properties that have been set (i.e. not the 
default).
* This creates a new copy of the config properties in the form of a Map.
@@ -3304,6 +3327,7 @@ class SQLConf extends Serializable with Logging {
* definition contains key, defaultValue and doc.
*/
   def getAllDefinedConfs: Seq[(String, String, String, String)] = 
sqlConfEntries.synchronized {
+loadDefinedConfs()
 sqlConfEntries.values.asScala.filter(_.isPublic).map { entry =>
   val displayValue = Option(getConfString(entry.key, 
null)).getOrElse(entry.defaultValueString)
   (entry.key, displayValue, entry.doc, entry.version)
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
index 3825460..2cd26e2 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
@@ -20,20 +20,17 @@ package org.apache.spark.sql.api.python
 import java.io.InputStream
 import

[spark] branch branch-3.1 updated: [SPARK-33438][SQL] Eagerly init objects with defined SQL Confs for command `set -v`

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 6ca8e75  [SPARK-33438][SQL] Eagerly init objects with defined SQL 
Confs for command `set -v`
6ca8e75 is described below

commit 6ca8e75592da52eba8b033a02aafa5045a69f8ed
Author: Linhong Liu 
AuthorDate: Mon Feb 8 22:48:28 2021 +0900

[SPARK-33438][SQL] Eagerly init objects with defined SQL Confs for command 
`set -v`

### What changes were proposed in this pull request?
In Spark, `set -v` is defined as "Queries all properties that are defined 
in the SQLConf of the sparkSession".
But there are other external modules that also define properties and 
register them to SQLConf. In this case,
it can't be displayed by `set -v` until the conf object is initiated (i.e. 
calling the object at least once).

In this PR, I propose to eagerly initiate all the objects registered to 
SQLConf, so that `set -v` will always output
the completed properties.

### Why are the changes needed?
Improve the `set -v` command to produces completed and  deterministic 
results

### Does this PR introduce _any_ user-facing change?
`set -v` command will dump more configs

### How was this patch tested?
existing tests

Closes #30363 from linhongliu-db/set-v.

Authored-by: Linhong Liu 
Signed-off-by: HyukjinKwon 
(cherry picked from commit 037bfb2dbcb73cfbd73f0fd9abe0b38789a182a2)
Signed-off-by: HyukjinKwon 
---
 .../org/apache/spark/sql/internal/SQLConf.scala| 24 ++
 .../spark/sql/api/python/PythonSQLUtils.scala  | 15 +-
 2 files changed, 25 insertions(+), 14 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 55c7016..ecd8f3a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -25,6 +25,7 @@ import java.util.zip.Deflater
 import scala.collection.JavaConverters._
 import scala.collection.immutable
 import scala.util.Try
+import scala.util.control.NonFatal
 import scala.util.matching.Regex
 
 import org.apache.hadoop.fs.Path
@@ -35,6 +36,7 @@ import org.apache.spark.internal.config._
 import org.apache.spark.internal.config.{IGNORE_MISSING_FILES => 
SPARK_IGNORE_MISSING_FILES}
 import org.apache.spark.network.util.ByteUnit
 import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.ScalaReflection
 import org.apache.spark.sql.catalyst.analysis.{HintErrorLogger, Resolver}
 import org.apache.spark.sql.catalyst.expressions.CodegenObjectFactoryMode
 import org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator
@@ -3768,6 +3770,27 @@ class SQLConf extends Serializable with Logging {
 }
   }
 
+  private var definedConfsLoaded = false
+  /**
+   * Init [[StaticSQLConf]] and [[org.apache.spark.sql.hive.HiveUtils]] so 
that all the defined
+   * SQL Configurations will be registered to SQLConf
+   */
+  private def loadDefinedConfs(): Unit = {
+if (!definedConfsLoaded) {
+  definedConfsLoaded = true
+  // Force to register static SQL configurations
+  StaticSQLConf
+  try {
+// Force to register SQL configurations from Hive module
+val symbol = 
ScalaReflection.mirror.staticModule("org.apache.spark.sql.hive.HiveUtils")
+ScalaReflection.mirror.reflectModule(symbol).instance
+  } catch {
+case NonFatal(e) =>
+  logWarning("SQL configurations from Hive module is not loaded", e)
+  }
+}
+  }
+
   /**
* Return all the configuration properties that have been set (i.e. not the 
default).
* This creates a new copy of the config properties in the form of a Map.
@@ -3780,6 +3803,7 @@ class SQLConf extends Serializable with Logging {
* definition contains key, defaultValue and doc.
*/
   def getAllDefinedConfs: Seq[(String, String, String, String)] = 
sqlConfEntries.synchronized {
+loadDefinedConfs()
 sqlConfEntries.values.asScala.filter(_.isPublic).map { entry =>
   val displayValue = Option(getConfString(entry.key, 
null)).getOrElse(entry.defaultValueString)
   (entry.key, displayValue, entry.doc, entry.version)
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
index 3825460..2cd26e2 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
@@ -20,20 +20,17 @@ package org.apache.spark.sql.api.python
 import java.io.InputStream
 import

[spark] branch master updated (a854906 -> 037bfb2)

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from a854906  [SPARK-34377][SQL] Add new parquet datasource options to 
control datetime rebasing in read
 add 037bfb2  [SPARK-33438][SQL] Eagerly init objects with defined SQL 
Confs for command `set -v`

No new revisions were added by this update.

Summary of changes:
 .../org/apache/spark/sql/internal/SQLConf.scala| 24 ++
 .../spark/sql/api/python/PythonSQLUtils.scala  | 15 +-
 2 files changed, 25 insertions(+), 14 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (70ef196 -> a854906)

2021-02-08 Thread wenchen 
This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 70ef196  [SPARK-34157][BUILD][FOLLOW-UP] Fix Scala 2.13 compilation 
error via using Array.deep
 add a854906  [SPARK-34377][SQL] Add new parquet datasource options to 
control datetime rebasing in read

No new revisions were added by this update.

Summary of changes:
 python/pyspark/sql/readwriter.py   | 29 -
 python/pyspark/sql/streaming.py| 30 -
 .../org/apache/spark/sql/DataFrameReader.scala | 23 ++
 .../datasources/parquet/ParquetFileFormat.scala|  7 +++-
 .../datasources/parquet/ParquetOptions.scala   | 25 +++
 .../v2/parquet/ParquetPartitionReaderFactory.scala | 10 +++--
 .../datasources/v2/parquet/ParquetScan.scala   | 15 +--
 .../spark/sql/streaming/DataStreamReader.scala | 23 ++
 .../parquet/ParquetRebaseDatetimeSuite.scala   | 49 +++---
 9 files changed, 185 insertions(+), 26 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (556ecd6 -> 70ef196)

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 556ecd6  [MINOR] Add a note about pip installation test in RC for 
release vote template
 add 70ef196  [SPARK-34157][BUILD][FOLLOW-UP] Fix Scala 2.13 compilation 
error via using Array.deep

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/command/v1/ShowTablesSuite.scala   | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (88ced28 -> 556ecd6)

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 88ced28  [SPARK-33354][DOC] Remove an unnecessary quote in doc
 add 556ecd6  [MINOR] Add a note about pip installation test in RC for 
release vote template

No new revisions were added by this update.

Summary of changes:
 dev/create-release/vote.tmpl | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.1 updated (76daa1f -> c4d90f3)

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 76daa1f  [MINOR][INFRA][DOC][3.1] Change the facetFilters of Docsearch 
to 3.1.1
 add cf0115a  Preparing Spark release v3.1.1-rc2
 new c4d90f3  Preparing development version 3.1.2-SNAPSHOT

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] 01/01: Preparing development version 3.1.2-SNAPSHOT

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git

commit c4d90f3d829523846f9d33d79503879fe86a8382
Author: Hyukjin Kwon 
AuthorDate: Mon Feb 8 13:10:08 2021 +

Preparing development version 3.1.2-SNAPSHOT
---
 R/pkg/DESCRIPTION  | 2 +-
 assembly/pom.xml   | 2 +-
 common/kvstore/pom.xml | 2 +-
 common/network-common/pom.xml  | 2 +-
 common/network-shuffle/pom.xml | 2 +-
 common/network-yarn/pom.xml| 2 +-
 common/sketch/pom.xml  | 2 +-
 common/tags/pom.xml| 2 +-
 common/unsafe/pom.xml  | 2 +-
 core/pom.xml   | 2 +-
 docs/_config.yml   | 4 ++--
 examples/pom.xml   | 2 +-
 external/avro/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml  | 2 +-
 external/kafka-0-10-assembly/pom.xml   | 2 +-
 external/kafka-0-10-sql/pom.xml| 2 +-
 external/kafka-0-10-token-provider/pom.xml | 2 +-
 external/kafka-0-10/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml  | 2 +-
 external/kinesis-asl/pom.xml   | 2 +-
 external/spark-ganglia-lgpl/pom.xml| 2 +-
 graphx/pom.xml | 2 +-
 hadoop-cloud/pom.xml   | 2 +-
 launcher/pom.xml   | 2 +-
 mllib-local/pom.xml| 2 +-
 mllib/pom.xml  | 2 +-
 pom.xml| 2 +-
 python/pyspark/version.py  | 2 +-
 repl/pom.xml   | 2 +-
 resource-managers/kubernetes/core/pom.xml  | 2 +-
 resource-managers/kubernetes/integration-tests/pom.xml | 2 +-
 resource-managers/mesos/pom.xml| 2 +-
 resource-managers/yarn/pom.xml | 2 +-
 sql/catalyst/pom.xml   | 2 +-
 sql/core/pom.xml   | 2 +-
 sql/hive-thriftserver/pom.xml  | 2 +-
 sql/hive/pom.xml   | 2 +-
 streaming/pom.xml  | 2 +-
 tools/pom.xml  | 2 +-
 39 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index 04ebb0c..9eb9cd3 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: SparkR
 Type: Package
-Version: 3.1.1
+Version: 3.1.2
 Title: R Front End for 'Apache Spark'
 Description: Provides an R Front end for 'Apache Spark' 
.
 Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 322f696..2a11234 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.1
+3.1.2-SNAPSHOT
 ../pom.xml
   
 
diff --git a/common/kvstore/pom.xml b/common/kvstore/pom.xml
index d6d454d..d65557f0 100644
--- a/common/kvstore/pom.xml
+++ b/common/kvstore/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.1
+3.1.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 49396ee..1cbab66 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.1
+3.1.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index c349871..e654675 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.1
+3.1.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 97b0b04..c49bbfa 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.1
+3.1.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/sketch/pom.xml b/common/sketch/pom.xml
index 6315452..7e4fdf2 100644
--- a/common/sketch/pom.xml
+++ b/common/sketch/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.1
+3.1.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/tags/pom.xml

[spark] 01/01: Preparing Spark release v3.1.1-rc2

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to tag v3.1.1-rc2
in repository https://gitbox.apache.org/repos/asf/spark.git

commit cf0115ac2d60070399af481b14566f33d22ec45e
Author: Hyukjin Kwon 
AuthorDate: Mon Feb 8 13:10:01 2021 +

Preparing Spark release v3.1.1-rc2
---
 R/pkg/DESCRIPTION  | 2 +-
 assembly/pom.xml   | 2 +-
 common/kvstore/pom.xml | 2 +-
 common/network-common/pom.xml  | 2 +-
 common/network-shuffle/pom.xml | 2 +-
 common/network-yarn/pom.xml| 2 +-
 common/sketch/pom.xml  | 2 +-
 common/tags/pom.xml| 2 +-
 common/unsafe/pom.xml  | 2 +-
 core/pom.xml   | 2 +-
 docs/_config.yml   | 4 ++--
 examples/pom.xml   | 2 +-
 external/avro/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml  | 2 +-
 external/kafka-0-10-assembly/pom.xml   | 2 +-
 external/kafka-0-10-sql/pom.xml| 2 +-
 external/kafka-0-10-token-provider/pom.xml | 2 +-
 external/kafka-0-10/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml  | 2 +-
 external/kinesis-asl/pom.xml   | 2 +-
 external/spark-ganglia-lgpl/pom.xml| 2 +-
 graphx/pom.xml | 2 +-
 hadoop-cloud/pom.xml   | 2 +-
 launcher/pom.xml   | 2 +-
 mllib-local/pom.xml| 2 +-
 mllib/pom.xml  | 2 +-
 pom.xml| 2 +-
 python/pyspark/version.py  | 2 +-
 repl/pom.xml   | 2 +-
 resource-managers/kubernetes/core/pom.xml  | 2 +-
 resource-managers/kubernetes/integration-tests/pom.xml | 2 +-
 resource-managers/mesos/pom.xml| 2 +-
 resource-managers/yarn/pom.xml | 2 +-
 sql/catalyst/pom.xml   | 2 +-
 sql/core/pom.xml   | 2 +-
 sql/hive-thriftserver/pom.xml  | 2 +-
 sql/hive/pom.xml   | 2 +-
 streaming/pom.xml  | 2 +-
 tools/pom.xml  | 2 +-
 39 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index 9eb9cd3..04ebb0c 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: SparkR
 Type: Package
-Version: 3.1.2
+Version: 3.1.1
 Title: R Front End for 'Apache Spark'
 Description: Provides an R Front end for 'Apache Spark' 
.
 Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 2a11234..322f696 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.2-SNAPSHOT
+3.1.1
 ../pom.xml
   
 
diff --git a/common/kvstore/pom.xml b/common/kvstore/pom.xml
index d65557f0..d6d454d 100644
--- a/common/kvstore/pom.xml
+++ b/common/kvstore/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.2-SNAPSHOT
+3.1.1
 ../../pom.xml
   
 
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 1cbab66..49396ee 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.2-SNAPSHOT
+3.1.1
 ../../pom.xml
   
 
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index e654675..c349871 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.2-SNAPSHOT
+3.1.1
 ../../pom.xml
   
 
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index c49bbfa..97b0b04 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.2-SNAPSHOT
+3.1.1
 ../../pom.xml
   
 
diff --git a/common/sketch/pom.xml b/common/sketch/pom.xml
index 7e4fdf2..6315452 100644
--- a/common/sketch/pom.xml
+++ b/common/sketch/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.1.2-SNAPSHOT
+3.1.1
 ../../pom.xml
   
 
diff --git a/common/tags/pom.xml b/common/tags/pom.xml
index

[spark] tag v3.1.1-rc2 created (now cf0115a)

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to tag v3.1.1-rc2
in repository https://gitbox.apache.org/repos/asf/spark.git.


  at cf0115a  (commit)
This tag includes the following new commits:

 new cf0115a  Preparing Spark release v3.1.1-rc2

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.1 updated (0ac4f04 -> 76daa1f)

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 0ac4f04  [SPARK-33354][DOC] Remove an unnecessary quote in doc
 add 76daa1f  [MINOR][INFRA][DOC][3.1] Change the facetFilters of Docsearch 
to 3.1.1

No new revisions were added by this update.

Summary of changes:
 docs/_config.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.1 updated: [SPARK-33354][DOC] Remove an unnecessary quote in doc

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 0ac4f04  [SPARK-33354][DOC] Remove an unnecessary quote in doc
0ac4f04 is described below

commit 0ac4f04b8bbce75526308e6a93b11f1ba8de77e7
Author: Gengliang Wang 
AuthorDate: Mon Feb 8 21:08:34 2021 +0900

[SPARK-33354][DOC] Remove an unnecessary quote in doc

Remove an unnecessary quote in the documentation.
Super trivial.

Fix a mistake.

No

Just doc

Closes #31523 from gengliangwang/removeQuote.

Authored-by: Gengliang Wang 
Signed-off-by: HyukjinKwon 
(cherry picked from commit 88ced28141beb696791ae67eac35219de942bf31)
Signed-off-by: HyukjinKwon 
---
 docs/sql-ref-ansi-compliance.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/sql-ref-ansi-compliance.md b/docs/sql-ref-ansi-compliance.md
index 983633a..bb62f76 100644
--- a/docs/sql-ref-ansi-compliance.md
+++ b/docs/sql-ref-ansi-compliance.md
@@ -62,7 +62,7 @@ Spark SQL has three kinds of type conversions: explicit 
casting, type coercion,
 When `spark.sql.ansi.enabled` is set to `true`, explicit casting by `CAST` 
syntax throws a runtime exception for illegal cast patterns defined in the 
standard, e.g. casts from a string to an integer.
 On the other hand, `INSERT INTO` syntax throws an analysis exception when the 
ANSI mode enabled via `spark.sql.storeAssignmentPolicy=ANSI`.
 
-The type conversion of Spark ANSI mode follows the syntax rules of section 
6.13 "cast specification" in [ISO/IEC 9075-2:2011 Information technology — 
Database languages - SQL — Part 2: Foundation 
(SQL/Foundation)"](https://www.iso.org/standard/53682.html), except it 
specially allows the following
+The type conversion of Spark ANSI mode follows the syntax rules of section 
6.13 "cast specification" in [ISO/IEC 9075-2:2011 Information technology — 
Database languages - SQL — Part 2: Foundation 
(SQL/Foundation)](https://www.iso.org/standard/53682.html), except it specially 
allows the following
  straightforward type conversions which are disallowed as per the ANSI 
standard:
 * NumericType <=> BooleanType
 * StringType <=> BinaryType


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated: [SPARK-33354][DOC] Remove an unnecessary quote in doc

2021-02-08 Thread gurwls223 
This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new 88ced28  [SPARK-33354][DOC] Remove an unnecessary quote in doc
88ced28 is described below

commit 88ced28141beb696791ae67eac35219de942bf31
Author: Gengliang Wang 
AuthorDate: Mon Feb 8 21:08:34 2021 +0900

[SPARK-33354][DOC] Remove an unnecessary quote in doc

### What changes were proposed in this pull request?

Remove an unnecessary quote in the documentation.
Super trivial.

### Why are the changes needed?

Fix a mistake.

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Just doc

Closes #31523 from gengliangwang/removeQuote.

Authored-by: Gengliang Wang 
Signed-off-by: HyukjinKwon 
---
 docs/sql-ref-ansi-compliance.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/sql-ref-ansi-compliance.md b/docs/sql-ref-ansi-compliance.md
index 022c27e..f97b166 100644
--- a/docs/sql-ref-ansi-compliance.md
+++ b/docs/sql-ref-ansi-compliance.md
@@ -62,7 +62,7 @@ Spark SQL has three kinds of type conversions: explicit 
casting, type coercion,
 When `spark.sql.ansi.enabled` is set to `true`, explicit casting by `CAST` 
syntax throws a runtime exception for illegal cast patterns defined in the 
standard, e.g. casts from a string to an integer.
 On the other hand, `INSERT INTO` syntax throws an analysis exception when the 
ANSI mode enabled via `spark.sql.storeAssignmentPolicy=ANSI`.
 
-The type conversion of Spark ANSI mode follows the syntax rules of section 
6.13 "cast specification" in [ISO/IEC 9075-2:2011 Information technology — 
Database languages - SQL — Part 2: Foundation 
(SQL/Foundation)"](https://www.iso.org/standard/53682.html), except it 
specially allows the following
+The type conversion of Spark ANSI mode follows the syntax rules of section 
6.13 "cast specification" in [ISO/IEC 9075-2:2011 Information technology — 
Database languages - SQL — Part 2: Foundation 
(SQL/Foundation)](https://www.iso.org/standard/53682.html), except it specially 
allows the following
  straightforward type conversions which are disallowed as per the ANSI 
standard:
 * NumericType <=> BooleanType
 * StringType <=> BinaryType


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (2c243c9 -> 70a79e9)

2021-02-08 Thread wenchen 
This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 2c243c9  [SPARK-34157][SQL] Unify output of SHOW TABLES and pass 
output attributes properly
 add 70a79e9  [SPARK-34239][SQL][FOLLOW_UP] SHOW COLUMNS Keep consistence 
with other `SHOW` command

No new revisions were added by this update.

Summary of changes:
 .../org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala | 9 +
 .../spark/sql/catalyst/analysis/ResolveSessionCatalog.scala  | 4 ++--
 .../sql/execution/datasources/v2/DataSourceV2Strategy.scala  | 2 +-
 3 files changed, 8 insertions(+), 7 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (9270238 -> 2c243c9)

2021-02-08 Thread wenchen 
This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 9270238  [SPARK-34355][SQL] Add log and time cost for commit job
 add 2c243c9  [SPARK-34157][SQL] Unify output of SHOW TABLES and pass 
output attributes properly

No new revisions were added by this update.

Summary of changes:
 docs/sql-migration-guide.md|  4 ++
 python/pyspark/sql/context.py  |  2 +-
 .../spark/sql/catalyst/analysis/Analyzer.scala |  4 +-
 .../sql/catalyst/plans/logical/v2Commands.scala| 17 +---
 .../scala/org/apache/spark/sql/SQLContext.scala|  7 ++--
 .../catalyst/analysis/ResolveSessionCatalog.scala  | 26 
 .../spark/sql/execution/command/tables.scala   | 14 +--
 .../datasources/v2/DataSourceV2Strategy.scala  |  4 +-
 .../execution/datasources/v2/ShowTablesExec.scala  |  2 +-
 .../execution/datasources/v2/V2CommandExec.scala   |  4 +-
 .../sql-tests/results/show-tables.sql.out  | 16 
 .../org/apache/spark/sql/SQLContextSuite.scala |  2 +-
 .../spark/sql/connector/DataSourceV2SQLSuite.scala |  4 +-
 .../execution/command/ShowTablesSuiteBase.scala| 32 ++-
 .../sql/execution/command/v1/ShowTablesSuite.scala | 46 +-
 .../sql/execution/command/v2/ShowTablesSuite.scala | 15 +--
 .../v2/jdbc/JDBCTableCatalogSuite.scala| 13 +++---
 .../org/apache/spark/sql/jdbc/JDBCV2Suite.scala|  2 +-
 18 files changed, 108 insertions(+), 106 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org