[spark] branch master updated (423ba5a -> 0046222)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 423ba5a [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove the newly added YarnShuffleServiceSuite.java add 0046222 [SPARK-7][SQL][FOLLOWUP] Prevent possible flakyness in SubexpressionEliminationSuite No new revisions were added by this update. Summary of changes: .../expressions/SubexpressionEliminationSuite.scala | 13 +++-- 1 file changed, 7 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (423ba5a -> 0046222)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 423ba5a [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove the newly added YarnShuffleServiceSuite.java add 0046222 [SPARK-7][SQL][FOLLOWUP] Prevent possible flakyness in SubexpressionEliminationSuite No new revisions were added by this update. Summary of changes: .../expressions/SubexpressionEliminationSuite.scala | 13 +++-- 1 file changed, 7 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (423ba5a -> 0046222)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 423ba5a [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove the newly added YarnShuffleServiceSuite.java add 0046222 [SPARK-7][SQL][FOLLOWUP] Prevent possible flakyness in SubexpressionEliminationSuite No new revisions were added by this update. Summary of changes: .../expressions/SubexpressionEliminationSuite.scala | 13 +++-- 1 file changed, 7 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (423ba5a -> 0046222)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 423ba5a [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove the newly added YarnShuffleServiceSuite.java add 0046222 [SPARK-7][SQL][FOLLOWUP] Prevent possible flakyness in SubexpressionEliminationSuite No new revisions were added by this update. Summary of changes: .../expressions/SubexpressionEliminationSuite.scala | 13 +++-- 1 file changed, 7 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (423ba5a -> 0046222)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 423ba5a [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove the newly added YarnShuffleServiceSuite.java add 0046222 [SPARK-7][SQL][FOLLOWUP] Prevent possible flakyness in SubexpressionEliminationSuite No new revisions were added by this update. Summary of changes: .../expressions/SubexpressionEliminationSuite.scala | 13 +++-- 1 file changed, 7 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove the newly added YarnShuffleServiceSuite.java
This is an automated email from the ASF dual-hosted git repository. tgraves pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 423ba5a [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove the newly added YarnShuffleServiceSuite.java 423ba5a is described below commit 423ba5a16038c1cb28d0973e18518645e69d5ff1 Author: Chandni Singh AuthorDate: Fri Nov 13 16:16:23 2020 -0600 [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove the newly added YarnShuffleServiceSuite.java ### What changes were proposed in this pull request? This is a follow-up fix for the failing tests in `YarnShuffleServiceSuite.java`. This java class was introduced in https://github.com/apache/spark/pull/30062. The tests in the class fail when run with hadoop-2.7 profile: ``` [ERROR] testCreateDefaultMergedShuffleFileManagerInstance(org.apache.spark.network.yarn.YarnShuffleServiceSuite) Time elapsed: 0.627 s <<< ERROR! java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory at org.apache.spark.network.yarn.YarnShuffleServiceSuite.testCreateDefaultMergedShuffleFileManagerInstance(YarnShuffleServiceSuite.java:37) Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory at org.apache.spark.network.yarn.YarnShuffleServiceSuite.testCreateDefaultMergedShuffleFileManagerInstance(YarnShuffleServiceSuite.java:37) [ERROR] testCreateRemoteBlockPushResolverInstance(org.apache.spark.network.yarn.YarnShuffleServiceSuite) Time elapsed: 0 s <<< ERROR! java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.network.yarn.YarnShuffleService at org.apache.spark.network.yarn.YarnShuffleServiceSuite.testCreateRemoteBlockPushResolverInstance(YarnShuffleServiceSuite.java:47) [ERROR] testInvalidClassNameOfMergeManagerWillUseNoOpInstance(org.apache.spark.network.yarn.YarnShuffleServiceSuite) Time elapsed: 0.001 s <<< ERROR! java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.network.yarn.YarnShuffleService at org.apache.spark.network.yarn.YarnShuffleServiceSuite.testInvalidClassNameOfMergeManagerWillUseNoOpInstance(YarnShuffleServiceSuite.java:57) ``` A test suit for `YarnShuffleService` did exist here: `resource-managers/yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceSuite.scala` I missed this when I created `common/network-yarn/src/test/java/org/apache/spark/network/yarn/YarnShuffleServiceSuite.java`. Moving all the new tests to the earlier test suite fixes the failures with hadoop-2.7 even though why this happened is not clear. ### Why are the changes needed? The newly added tests are failing when run with hadoop profile 2.7 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Ran the unit tests with the default profile as well as hadoop 2.7 profile. `build/mvn test -Dtest=none -DwildcardSuites=org.apache.spark.network.yarn.YarnShuffleServiceSuite -Phadoop-2.7 -Pyarn` ``` Run starting. Expected test count is: 11 YarnShuffleServiceSuite: - executor state kept across NM restart - removed applications should not be in registered executor file - shuffle service should be robust to corrupt registered executor file - get correct recovery path - moving recovery file from NM local dir to recovery path - service throws error if cannot start - recovery db should not be created if NM recovery is not enabled - SPARK-31646: metrics should be registered into Node Manager's metrics system - create default merged shuffle file manager instance - create remote block push resolver instance - invalid class name of merge manager will use noop instance Run completed in 2 seconds, 572 milliseconds. Total number of tests run: 11 Suites: completed 2, aborted 0 Tests: succeeded 11, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #30349 from otterc/SPARK-32916-followup. Authored-by: Chandni Singh Signed-off-by: Thomas Graves --- .../network/yarn/YarnShuffleServiceSuite.java | 61 -- .../network/yarn/YarnShuffleServiceSuite.scala | 27 +- 2 files changed, 26 insertions(+), 62 deletions(-) diff --git a/common/network-yarn/src/test/java/org/apache/spark/network/yarn/YarnShuffleServiceSuite.java b/common/network-yarn/src/test/java/org/apache/spark/network/yarn/YarnShuffleServiceSuite.java deleted file mode 100644 index 09bc4d8..000 --- a/common/network-yarn/src/test/java/org/apache/spark/network/yarn/YarnShuffleServiceSuite.java +++ /dev/null @@ -1,61 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See
[spark] branch master updated: [SPARK-33288][SPARK-32661][K8S] Stage level scheduling support for Kubernetes
This is an automated email from the ASF dual-hosted git repository.
tgraves pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new acfd846 [SPARK-33288][SPARK-32661][K8S] Stage level scheduling
support for Kubernetes
acfd846 is described below
commit acfd8467534fbf58c12e9f2d993b7d135fb8d32b
Author: Thomas Graves
AuthorDate: Fri Nov 13 16:04:13 2020 -0600
[SPARK-33288][SPARK-32661][K8S] Stage level scheduling support for
Kubernetes
### What changes were proposed in this pull request?
This adds support for Stage level scheduling to kubernetes. Kubernetes can
support dynamic allocation via the shuffle tracking option which means we can
support stage level scheduling by getting new executors.
The main changes here are having the k8s cluster manager pass the resource
profile id into the executors and then the ExecutorsPodsAllocator has to
request executors based on the individual resource profiles. I tried to keep
code changes here to a minimum. I specifically choose to leave the
ExecutorPodsSnapshot the way it was and construct the resource profile to pod
states on the fly, with a fast path when not using other resource profiles, to
keep the impact to a minimum. This res [...]
This also adds support for [SPARK-32661]Spark executors on K8S should
request extra memory for off-heap allocations because the stage level
scheduling api has support for this and it made sense to make consistent with
YARN. This was started with PR https://github.com/apache/spark/pull/29477 but
never updated so I just did it here. To do this I moved a few functions
around that were now used by both YARN and kubernetes so you will see some
changes in Utils.
### Why are the changes needed?
Add the feature to Kubernetes based on customer feedback.
### Does this PR introduce _any_ user-facing change?
Yes the feature now works with K8s, but not underlying API changes.
### How was this patch tested?
Tested manually on kubernetes cluster and with unit tests.
Closes #30204 from tgravescs/stagek8sOrigSnapshotsRebase.
Lead-authored-by: Thomas Graves
Co-authored-by: Thomas Graves
Signed-off-by: Thomas Graves
---
.../apache/spark/resource/ResourceProfile.scala| 133 +-
.../spark/resource/ResourceProfileManager.scala| 21 +-
.../main/scala/org/apache/spark/util/Utils.scala | 21 ++
.../resource/ResourceProfileManagerSuite.scala | 27 +-
.../spark/resource/ResourceProfileSuite.scala | 8 +-
.../scala/org/apache/spark/util/UtilsSuite.scala | 27 ++
docs/configuration.md | 2 +-
docs/running-on-kubernetes.md | 4 +
docs/running-on-yarn.md| 1 +
.../org/apache/spark/deploy/k8s/Constants.scala| 3 +-
.../apache/spark/deploy/k8s/KubernetesConf.scala | 12 +-
.../k8s/features/BasicDriverFeatureStep.scala | 4 +-
.../k8s/features/BasicExecutorFeatureStep.scala| 89 ---
.../cluster/k8s/ExecutorPodsAllocator.scala| 285 -
.../k8s/KubernetesClusterSchedulerBackend.scala| 5 +-
.../cluster/k8s/KubernetesExecutorBuilder.scala| 8 +-
.../spark/deploy/k8s/KubernetesConfSuite.scala | 15 +-
.../k8s/features/BasicDriverFeatureStepSuite.scala | 5 +-
.../features/BasicExecutorFeatureStepSuite.scala | 106 ++--
.../cluster/k8s/ExecutorLifecycleTestUtils.scala | 40 +--
.../cluster/k8s/ExecutorPodsAllocatorSuite.scala | 109 +++-
.../KubernetesClusterSchedulerBackendSuite.scala | 5 +-
.../k8s/KubernetesExecutorBuilderSuite.scala | 4 +-
.../src/main/dockerfiles/spark/entrypoint.sh | 1 +
.../org/apache/spark/deploy/yarn/Client.scala | 9 +-
.../spark/deploy/yarn/ResourceRequestHelper.scala | 4 +-
.../apache/spark/deploy/yarn/YarnAllocator.scala | 136 +-
.../spark/deploy/yarn/YarnSparkHadoopUtil.scala| 24 +-
.../spark/deploy/yarn/YarnAllocatorSuite.scala | 20 +-
.../deploy/yarn/YarnSparkHadoopUtilSuite.scala | 27 --
30 files changed, 772 insertions(+), 383 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
b/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
index 8a37670..ac7e8e8 100644
--- a/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
+++ b/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
@@ -29,6 +29,7 @@ import org.apache.spark.annotation.{Evolving, Since}
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.Python.PYSPARK_EXECUTOR_MEMORY
+import org.apache.spark.util.Utils
/**
* Resource profile to associate with an RDD. A ResourceProfile allows the
user to
@@
[spark] branch master updated (f80fe21 -> 234711a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f80fe21 [SPARK-33166][DOC] Provide Search Function in Spark docs site add 234711a Revert "[SPARK-33139][SQL] protect setActionSession and clearActiveSession" No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md| 2 -- .../sql/kafka010/KafkaMicroBatchSourceSuite.scala | 2 +- .../java/org/apache/spark/SharedSparkSession.java | 3 +- .../spark/mllib/util/MLlibTestSparkContext.scala | 2 +- python/pyspark/sql/session.py | 8 ++--- .../org/apache/spark/sql/internal/SQLConf.scala| 3 -- .../apache/spark/sql/internal/StaticSQLConf.scala | 9 -- .../scala/org/apache/spark/sql/SparkSession.scala | 26 +++ .../apache/spark/sql/execution/SQLExecution.scala | 6 ++-- .../org/apache/spark/sql/execution/SparkPlan.scala | 2 +- .../sql/execution/streaming/StreamExecution.scala | 2 +- .../org/apache/spark/sql/DeprecatedAPISuite.scala | 6 ++-- .../org/apache/spark/sql/LocalSparkSession.scala | 4 +-- .../org/apache/spark/sql/SQLContextSuite.scala | 2 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 2 +- .../org/apache/spark/sql/SessionStateSuite.scala | 2 +- .../spark/sql/SparkSessionBuilderSuite.scala | 37 ++ .../spark/sql/SparkSessionExtensionSuite.scala | 2 +- .../spark/sql/connector/V1WriteFallbackSuite.scala | 4 +-- .../execution/CoalesceShufflePartitionsSuite.scala | 4 +-- .../adaptive/AdaptiveQueryExecSuite.scala | 4 +-- .../state/StateStoreCoordinatorSuite.scala | 2 +- .../streaming/state/StateStoreSuite.scala | 2 +- .../state/SymmetricHashJoinStateManagerSuite.scala | 2 +- .../spark/sql/streaming/StreamingJoinSuite.scala | 4 +-- .../org/apache/spark/sql/test/SQLTestUtils.scala | 2 +- .../apache/spark/sql/test/SharedSparkSession.scala | 2 +- .../org/apache/spark/sql/test/TestSQLContext.scala | 2 +- .../sql/hive/thriftserver/SparkOperation.scala | 6 ++-- .../spark/sql/hive/HiveSharedStateSuite.scala | 2 +- .../org/apache/spark/sql/hive/test/TestHive.scala | 2 +- 31 files changed, 52 insertions(+), 106 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f80fe21 -> 234711a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f80fe21 [SPARK-33166][DOC] Provide Search Function in Spark docs site add 234711a Revert "[SPARK-33139][SQL] protect setActionSession and clearActiveSession" No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md| 2 -- .../sql/kafka010/KafkaMicroBatchSourceSuite.scala | 2 +- .../java/org/apache/spark/SharedSparkSession.java | 3 +- .../spark/mllib/util/MLlibTestSparkContext.scala | 2 +- python/pyspark/sql/session.py | 8 ++--- .../org/apache/spark/sql/internal/SQLConf.scala| 3 -- .../apache/spark/sql/internal/StaticSQLConf.scala | 9 -- .../scala/org/apache/spark/sql/SparkSession.scala | 26 +++ .../apache/spark/sql/execution/SQLExecution.scala | 6 ++-- .../org/apache/spark/sql/execution/SparkPlan.scala | 2 +- .../sql/execution/streaming/StreamExecution.scala | 2 +- .../org/apache/spark/sql/DeprecatedAPISuite.scala | 6 ++-- .../org/apache/spark/sql/LocalSparkSession.scala | 4 +-- .../org/apache/spark/sql/SQLContextSuite.scala | 2 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 2 +- .../org/apache/spark/sql/SessionStateSuite.scala | 2 +- .../spark/sql/SparkSessionBuilderSuite.scala | 37 ++ .../spark/sql/SparkSessionExtensionSuite.scala | 2 +- .../spark/sql/connector/V1WriteFallbackSuite.scala | 4 +-- .../execution/CoalesceShufflePartitionsSuite.scala | 4 +-- .../adaptive/AdaptiveQueryExecSuite.scala | 4 +-- .../state/StateStoreCoordinatorSuite.scala | 2 +- .../streaming/state/StateStoreSuite.scala | 2 +- .../state/SymmetricHashJoinStateManagerSuite.scala | 2 +- .../spark/sql/streaming/StreamingJoinSuite.scala | 4 +-- .../org/apache/spark/sql/test/SQLTestUtils.scala | 2 +- .../apache/spark/sql/test/SharedSparkSession.scala | 2 +- .../org/apache/spark/sql/test/TestSQLContext.scala | 2 +- .../sql/hive/thriftserver/SparkOperation.scala | 6 ++-- .../spark/sql/hive/HiveSharedStateSuite.scala | 2 +- .../org/apache/spark/sql/hive/test/TestHive.scala | 2 +- 31 files changed, 52 insertions(+), 106 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f80fe21 -> 234711a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f80fe21 [SPARK-33166][DOC] Provide Search Function in Spark docs site add 234711a Revert "[SPARK-33139][SQL] protect setActionSession and clearActiveSession" No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md| 2 -- .../sql/kafka010/KafkaMicroBatchSourceSuite.scala | 2 +- .../java/org/apache/spark/SharedSparkSession.java | 3 +- .../spark/mllib/util/MLlibTestSparkContext.scala | 2 +- python/pyspark/sql/session.py | 8 ++--- .../org/apache/spark/sql/internal/SQLConf.scala| 3 -- .../apache/spark/sql/internal/StaticSQLConf.scala | 9 -- .../scala/org/apache/spark/sql/SparkSession.scala | 26 +++ .../apache/spark/sql/execution/SQLExecution.scala | 6 ++-- .../org/apache/spark/sql/execution/SparkPlan.scala | 2 +- .../sql/execution/streaming/StreamExecution.scala | 2 +- .../org/apache/spark/sql/DeprecatedAPISuite.scala | 6 ++-- .../org/apache/spark/sql/LocalSparkSession.scala | 4 +-- .../org/apache/spark/sql/SQLContextSuite.scala | 2 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 2 +- .../org/apache/spark/sql/SessionStateSuite.scala | 2 +- .../spark/sql/SparkSessionBuilderSuite.scala | 37 ++ .../spark/sql/SparkSessionExtensionSuite.scala | 2 +- .../spark/sql/connector/V1WriteFallbackSuite.scala | 4 +-- .../execution/CoalesceShufflePartitionsSuite.scala | 4 +-- .../adaptive/AdaptiveQueryExecSuite.scala | 4 +-- .../state/StateStoreCoordinatorSuite.scala | 2 +- .../streaming/state/StateStoreSuite.scala | 2 +- .../state/SymmetricHashJoinStateManagerSuite.scala | 2 +- .../spark/sql/streaming/StreamingJoinSuite.scala | 4 +-- .../org/apache/spark/sql/test/SQLTestUtils.scala | 2 +- .../apache/spark/sql/test/SharedSparkSession.scala | 2 +- .../org/apache/spark/sql/test/TestSQLContext.scala | 2 +- .../sql/hive/thriftserver/SparkOperation.scala | 6 ++-- .../spark/sql/hive/HiveSharedStateSuite.scala | 2 +- .../org/apache/spark/sql/hive/test/TestHive.scala | 2 +- 31 files changed, 52 insertions(+), 106 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f80fe21 -> 234711a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f80fe21 [SPARK-33166][DOC] Provide Search Function in Spark docs site add 234711a Revert "[SPARK-33139][SQL] protect setActionSession and clearActiveSession" No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md| 2 -- .../sql/kafka010/KafkaMicroBatchSourceSuite.scala | 2 +- .../java/org/apache/spark/SharedSparkSession.java | 3 +- .../spark/mllib/util/MLlibTestSparkContext.scala | 2 +- python/pyspark/sql/session.py | 8 ++--- .../org/apache/spark/sql/internal/SQLConf.scala| 3 -- .../apache/spark/sql/internal/StaticSQLConf.scala | 9 -- .../scala/org/apache/spark/sql/SparkSession.scala | 26 +++ .../apache/spark/sql/execution/SQLExecution.scala | 6 ++-- .../org/apache/spark/sql/execution/SparkPlan.scala | 2 +- .../sql/execution/streaming/StreamExecution.scala | 2 +- .../org/apache/spark/sql/DeprecatedAPISuite.scala | 6 ++-- .../org/apache/spark/sql/LocalSparkSession.scala | 4 +-- .../org/apache/spark/sql/SQLContextSuite.scala | 2 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 2 +- .../org/apache/spark/sql/SessionStateSuite.scala | 2 +- .../spark/sql/SparkSessionBuilderSuite.scala | 37 ++ .../spark/sql/SparkSessionExtensionSuite.scala | 2 +- .../spark/sql/connector/V1WriteFallbackSuite.scala | 4 +-- .../execution/CoalesceShufflePartitionsSuite.scala | 4 +-- .../adaptive/AdaptiveQueryExecSuite.scala | 4 +-- .../state/StateStoreCoordinatorSuite.scala | 2 +- .../streaming/state/StateStoreSuite.scala | 2 +- .../state/SymmetricHashJoinStateManagerSuite.scala | 2 +- .../spark/sql/streaming/StreamingJoinSuite.scala | 4 +-- .../org/apache/spark/sql/test/SQLTestUtils.scala | 2 +- .../apache/spark/sql/test/SharedSparkSession.scala | 2 +- .../org/apache/spark/sql/test/TestSQLContext.scala | 2 +- .../sql/hive/thriftserver/SparkOperation.scala | 6 ++-- .../spark/sql/hive/HiveSharedStateSuite.scala | 2 +- .../org/apache/spark/sql/hive/test/TestHive.scala | 2 +- 31 files changed, 52 insertions(+), 106 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f80fe21 -> 234711a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f80fe21 [SPARK-33166][DOC] Provide Search Function in Spark docs site add 234711a Revert "[SPARK-33139][SQL] protect setActionSession and clearActiveSession" No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md| 2 -- .../sql/kafka010/KafkaMicroBatchSourceSuite.scala | 2 +- .../java/org/apache/spark/SharedSparkSession.java | 3 +- .../spark/mllib/util/MLlibTestSparkContext.scala | 2 +- python/pyspark/sql/session.py | 8 ++--- .../org/apache/spark/sql/internal/SQLConf.scala| 3 -- .../apache/spark/sql/internal/StaticSQLConf.scala | 9 -- .../scala/org/apache/spark/sql/SparkSession.scala | 26 +++ .../apache/spark/sql/execution/SQLExecution.scala | 6 ++-- .../org/apache/spark/sql/execution/SparkPlan.scala | 2 +- .../sql/execution/streaming/StreamExecution.scala | 2 +- .../org/apache/spark/sql/DeprecatedAPISuite.scala | 6 ++-- .../org/apache/spark/sql/LocalSparkSession.scala | 4 +-- .../org/apache/spark/sql/SQLContextSuite.scala | 2 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 2 +- .../org/apache/spark/sql/SessionStateSuite.scala | 2 +- .../spark/sql/SparkSessionBuilderSuite.scala | 37 ++ .../spark/sql/SparkSessionExtensionSuite.scala | 2 +- .../spark/sql/connector/V1WriteFallbackSuite.scala | 4 +-- .../execution/CoalesceShufflePartitionsSuite.scala | 4 +-- .../adaptive/AdaptiveQueryExecSuite.scala | 4 +-- .../state/StateStoreCoordinatorSuite.scala | 2 +- .../streaming/state/StateStoreSuite.scala | 2 +- .../state/SymmetricHashJoinStateManagerSuite.scala | 2 +- .../spark/sql/streaming/StreamingJoinSuite.scala | 4 +-- .../org/apache/spark/sql/test/SQLTestUtils.scala | 2 +- .../apache/spark/sql/test/SharedSparkSession.scala | 2 +- .../org/apache/spark/sql/test/TestSQLContext.scala | 2 +- .../sql/hive/thriftserver/SparkOperation.scala | 6 ++-- .../spark/sql/hive/HiveSharedStateSuite.scala | 2 +- .../org/apache/spark/sql/hive/test/TestHive.scala | 2 +- 31 files changed, 52 insertions(+), 106 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cdd8e51 -> f80fe21)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from cdd8e51 [SPARK-33419][SQL] Unexpected behavior when using SET commands before a query in SparkSession.sql add f80fe21 [SPARK-33166][DOC] Provide Search Function in Spark docs site No new revisions were added by this update. Summary of changes: docs/_layouts/global.html | 23 +++ docs/css/docsearch.css| 36 2 files changed, 59 insertions(+) create mode 100644 docs/css/docsearch.css - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cdd8e51 -> f80fe21)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from cdd8e51 [SPARK-33419][SQL] Unexpected behavior when using SET commands before a query in SparkSession.sql add f80fe21 [SPARK-33166][DOC] Provide Search Function in Spark docs site No new revisions were added by this update. Summary of changes: docs/_layouts/global.html | 23 +++ docs/css/docsearch.css| 36 2 files changed, 59 insertions(+) create mode 100644 docs/css/docsearch.css - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cdd8e51 -> f80fe21)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from cdd8e51 [SPARK-33419][SQL] Unexpected behavior when using SET commands before a query in SparkSession.sql add f80fe21 [SPARK-33166][DOC] Provide Search Function in Spark docs site No new revisions were added by this update. Summary of changes: docs/_layouts/global.html | 23 +++ docs/css/docsearch.css| 36 2 files changed, 59 insertions(+) create mode 100644 docs/css/docsearch.css - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cdd8e51 -> f80fe21)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from cdd8e51 [SPARK-33419][SQL] Unexpected behavior when using SET commands before a query in SparkSession.sql add f80fe21 [SPARK-33166][DOC] Provide Search Function in Spark docs site No new revisions were added by this update. Summary of changes: docs/_layouts/global.html | 23 +++ docs/css/docsearch.css| 36 2 files changed, 59 insertions(+) create mode 100644 docs/css/docsearch.css - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
