[spark] branch master updated (b344e91 -> 9270238)
This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b344e91 [SPARK-34375][CORE][K8S][TEST] Replaces 'Mockito.initMocks' with 'Mockito.openMocks' add 9270238 [SPARK-34355][SQL] Add log and time cost for commit job No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/datasources/FileFormatWriter.scala| 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (329f945 -> b344e91)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 329f945 [SPARK-34391][BUILD] Upgrade commons-io to 2.8.0 add b344e91 [SPARK-34375][CORE][K8S][TEST] Replaces 'Mockito.initMocks' with 'Mockito.openMocks' No new revisions were added by this update. Summary of changes: .../java/org/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java | 4 ++-- .../org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java | 4 ++-- .../spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java | 4 ++-- core/src/test/scala/org/apache/spark/deploy/worker/WorkerSuite.scala | 2 +- .../test/scala/org/apache/spark/shuffle/ShuffleBlockPusherSuite.scala | 2 +- .../apache/spark/shuffle/sort/BypassMergeSortShuffleWriterSuite.scala | 2 +- .../apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala| 2 +- .../scala/org/apache/spark/shuffle/sort/SortShuffleWriterSuite.scala | 2 +- .../spark/shuffle/sort/io/LocalDiskShuffleMapOutputWriterSuite.scala | 2 +- .../test/scala/org/apache/spark/deploy/k8s/submit/ClientSuite.scala | 2 +- .../scala/org/apache/spark/deploy/k8s/submit/K8sSubmitOpSuite.scala | 2 +- .../spark/scheduler/cluster/k8s/ExecutorPodsAllocatorSuite.scala | 2 +- .../scheduler/cluster/k8s/ExecutorPodsLifecycleManagerSuite.scala | 2 +- .../cluster/k8s/ExecutorPodsPollingSnapshotSourceSuite.scala | 2 +- .../scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSourceSuite.scala | 2 +- .../cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala | 2 +- 16 files changed, 19 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (dcaf62a -> 329f945)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from dcaf62a [SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml add 329f945 [SPARK-34391][BUILD] Upgrade commons-io to 2.8.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 579e158 [SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing
core-site.xml
579e158 is described below
commit 579e15873c5a8ff3fcaafec5ce99e37884768f51
Author: Dongjoon Hyun
AuthorDate: Mon Feb 8 11:32:23 2021 +0900
[SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
### What changes were proposed in this pull request?
This is a follow-up for SPARK-34346 which causes a flakiness due to
`core-site.xml` test resource file addition. This PR aims to remove the test
resource `core/src/test/resources/core-site.xml` from `core` module.
### Why are the changes needed?
Due to the test resource `core-site.xml`, YARN UT becomes flaky in GitHub
Action and Jenkins.
```
$ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SPARK-16414" -Pyarn
...
[info] YarnClusterSuite:
[info] - yarn-cluster should respect conf overrides in SparkHadoopUtil
(SPARK-16414, SPARK-23630) *** FAILED *** (20 seconds, 209 milliseconds)
[info] FAILED did not equal FINISHED (stdout/stderr was not captured)
(BaseYarnClusterSuite.scala:210)
```
To isolate more, we may use `SPARK_TEST_HADOOP_CONF_DIR` like `yarn`
module's `yarn/Client`, but it seems an overkill in `core` module.
```
// SPARK-23630: during testing, Spark scripts filter out hadoop conf dirs
so that user's
// environments do not interfere with tests. This allows a special env
variable during
// tests so that custom conf dirs can be used by unit tests.
val confDirs = Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR") ++
(if (Utils.isTesting) Seq("SPARK_TEST_HADOOP_CONF_DIR") else Nil)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #31515 from dongjoon-hyun/SPARK-34346-2.
Authored-by: Dongjoon Hyun
Signed-off-by: HyukjinKwon
(cherry picked from commit dcaf62afea8791e49a44c2062fe14bafdcc0e92f)
Signed-off-by: HyukjinKwon
---
core/src/test/resources/core-site.xml | 24 --
.../scala/org/apache/spark/SparkContextSuite.scala | 3 ++-
2 files changed, 2 insertions(+), 25 deletions(-)
diff --git a/core/src/test/resources/core-site.xml
b/core/src/test/resources/core-site.xml
deleted file mode 100644
index 84eddf8..000
--- a/core/src/test/resources/core-site.xml
+++ /dev/null
@@ -1,24 +0,0 @@
-
-
-
-
-hadoop.tmp.dir
-/tmp/hive_zero
-default is /tmp/hadoop-${user.name} and will be
overridden
-
-
diff --git a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
index e0a5f52..01b1deb 100644
--- a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
@@ -979,11 +979,12 @@ class SparkContextSuite extends SparkFunSuite with
LocalSparkContext with Eventu
val testKey = "hadoop.tmp.dir"
val bufferKey = "io.file.buffer.size"
val hadoopConf0 = new Configuration()
+hadoopConf0.set(testKey, "/tmp/hive_zero")
val hiveConfFile =
Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
assert(hiveConfFile != null)
hadoopConf0.addResource(hiveConfFile)
-assert(hadoopConf0.get(testKey) === "/tmp/hive_one")
+assert(hadoopConf0.get(testKey) === "/tmp/hive_zero")
assert(hadoopConf0.get(bufferKey) === "201811")
val sparkConf = new SparkConf()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 2fe20ef [SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing
core-site.xml
2fe20ef is described below
commit 2fe20ef060a80aff1a20ef30f5e61992efd0fe50
Author: Dongjoon Hyun
AuthorDate: Mon Feb 8 11:32:23 2021 +0900
[SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
### What changes were proposed in this pull request?
This is a follow-up for SPARK-34346 which causes a flakiness due to
`core-site.xml` test resource file addition. This PR aims to remove the test
resource `core/src/test/resources/core-site.xml` from `core` module.
### Why are the changes needed?
Due to the test resource `core-site.xml`, YARN UT becomes flaky in GitHub
Action and Jenkins.
```
$ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SPARK-16414" -Pyarn
...
[info] YarnClusterSuite:
[info] - yarn-cluster should respect conf overrides in SparkHadoopUtil
(SPARK-16414, SPARK-23630) *** FAILED *** (20 seconds, 209 milliseconds)
[info] FAILED did not equal FINISHED (stdout/stderr was not captured)
(BaseYarnClusterSuite.scala:210)
```
To isolate more, we may use `SPARK_TEST_HADOOP_CONF_DIR` like `yarn`
module's `yarn/Client`, but it seems an overkill in `core` module.
```
// SPARK-23630: during testing, Spark scripts filter out hadoop conf dirs
so that user's
// environments do not interfere with tests. This allows a special env
variable during
// tests so that custom conf dirs can be used by unit tests.
val confDirs = Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR") ++
(if (Utils.isTesting) Seq("SPARK_TEST_HADOOP_CONF_DIR") else Nil)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #31515 from dongjoon-hyun/SPARK-34346-2.
Authored-by: Dongjoon Hyun
Signed-off-by: HyukjinKwon
(cherry picked from commit dcaf62afea8791e49a44c2062fe14bafdcc0e92f)
Signed-off-by: HyukjinKwon
---
core/src/test/resources/core-site.xml | 24 --
.../scala/org/apache/spark/SparkContextSuite.scala | 3 ++-
2 files changed, 2 insertions(+), 25 deletions(-)
diff --git a/core/src/test/resources/core-site.xml
b/core/src/test/resources/core-site.xml
deleted file mode 100644
index 84eddf8..000
--- a/core/src/test/resources/core-site.xml
+++ /dev/null
@@ -1,24 +0,0 @@
-
-
-
-
-hadoop.tmp.dir
-/tmp/hive_zero
-default is /tmp/hadoop-${user.name} and will be
overridden
-
-
diff --git a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
index 7751918..0c0a9b8 100644
--- a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
@@ -1040,11 +1040,12 @@ class SparkContextSuite extends SparkFunSuite with
LocalSparkContext with Eventu
val testKey = "hadoop.tmp.dir"
val bufferKey = "io.file.buffer.size"
val hadoopConf0 = new Configuration()
+hadoopConf0.set(testKey, "/tmp/hive_zero")
val hiveConfFile =
Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
assert(hiveConfFile != null)
hadoopConf0.addResource(hiveConfFile)
-assert(hadoopConf0.get(testKey) === "/tmp/hive_one")
+assert(hadoopConf0.get(testKey) === "/tmp/hive_zero")
assert(hadoopConf0.get(bufferKey) === "201811")
val sparkConf = new SparkConf()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new dcaf62a [SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing
core-site.xml
dcaf62a is described below
commit dcaf62afea8791e49a44c2062fe14bafdcc0e92f
Author: Dongjoon Hyun
AuthorDate: Mon Feb 8 11:32:23 2021 +0900
[SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
### What changes were proposed in this pull request?
This is a follow-up for SPARK-34346 which causes a flakiness due to
`core-site.xml` test resource file addition. This PR aims to remove the test
resource `core/src/test/resources/core-site.xml` from `core` module.
### Why are the changes needed?
Due to the test resource `core-site.xml`, YARN UT becomes flaky in GitHub
Action and Jenkins.
```
$ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SPARK-16414" -Pyarn
...
[info] YarnClusterSuite:
[info] - yarn-cluster should respect conf overrides in SparkHadoopUtil
(SPARK-16414, SPARK-23630) *** FAILED *** (20 seconds, 209 milliseconds)
[info] FAILED did not equal FINISHED (stdout/stderr was not captured)
(BaseYarnClusterSuite.scala:210)
```
To isolate more, we may use `SPARK_TEST_HADOOP_CONF_DIR` like `yarn`
module's `yarn/Client`, but it seems an overkill in `core` module.
```
// SPARK-23630: during testing, Spark scripts filter out hadoop conf dirs
so that user's
// environments do not interfere with tests. This allows a special env
variable during
// tests so that custom conf dirs can be used by unit tests.
val confDirs = Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR") ++
(if (Utils.isTesting) Seq("SPARK_TEST_HADOOP_CONF_DIR") else Nil)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #31515 from dongjoon-hyun/SPARK-34346-2.
Authored-by: Dongjoon Hyun
Signed-off-by: HyukjinKwon
---
core/src/test/resources/core-site.xml | 24 --
.../scala/org/apache/spark/SparkContextSuite.scala | 3 ++-
2 files changed, 2 insertions(+), 25 deletions(-)
diff --git a/core/src/test/resources/core-site.xml
b/core/src/test/resources/core-site.xml
deleted file mode 100644
index 84eddf8..000
--- a/core/src/test/resources/core-site.xml
+++ /dev/null
@@ -1,24 +0,0 @@
-
-
-
-
-hadoop.tmp.dir
-/tmp/hive_zero
-default is /tmp/hadoop-${user.name} and will be
overridden
-
-
diff --git a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
index 4319761..7a4970e 100644
--- a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
@@ -1155,11 +1155,12 @@ class SparkContextSuite extends SparkFunSuite with
LocalSparkContext with Eventu
val testKey = "hadoop.tmp.dir"
val bufferKey = "io.file.buffer.size"
val hadoopConf0 = new Configuration()
+hadoopConf0.set(testKey, "/tmp/hive_zero")
val hiveConfFile =
Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
assert(hiveConfFile != null)
hadoopConf0.addResource(hiveConfFile)
-assert(hadoopConf0.get(testKey) === "/tmp/hive_one")
+assert(hadoopConf0.get(testKey) === "/tmp/hive_zero")
assert(hadoopConf0.get(bufferKey) === "201811")
val sparkConf = new SparkConf()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-34158] Incorrect url of the only developer Matei in pom.xml
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new 14d0419 [SPARK-34158] Incorrect url of the only developer Matei in pom.xml 14d0419 is described below commit 14d041986759c0721420b2c1f343fb00b7b31515 Author: Kevin AuthorDate: Mon Feb 8 09:15:05 2021 +0900 [SPARK-34158] Incorrect url of the only developer Matei in pom.xml ### What changes were proposed in this pull request? Update the Incorrect URL of the only developer Matei in pom.xml ### Why are the changes needed? The current link was broken ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? change the link to https://cs.stanford.edu/people/matei/ Closes #31512 from pingsutw/SPARK-34158. Authored-by: Kevin Signed-off-by: HyukjinKwon (cherry picked from commit 30ef3d6e1c00bd1f28e71511576daad223ba8b22) Signed-off-by: HyukjinKwon --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 68eea2f..9811826 100644 --- a/pom.xml +++ b/pom.xml @@ -48,7 +48,7 @@ matei Matei Zaharia matei.zaha...@gmail.com - http://www.cs.berkeley.edu/~matei + https://cs.stanford.edu/people/matei Apache Software Foundation http://spark.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-34158] Incorrect url of the only developer Matei in pom.xml
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 30ef3d6 [SPARK-34158] Incorrect url of the only developer Matei in pom.xml 30ef3d6 is described below commit 30ef3d6e1c00bd1f28e71511576daad223ba8b22 Author: Kevin AuthorDate: Mon Feb 8 09:15:05 2021 +0900 [SPARK-34158] Incorrect url of the only developer Matei in pom.xml ### What changes were proposed in this pull request? Update the Incorrect URL of the only developer Matei in pom.xml ### Why are the changes needed? The current link was broken ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? change the link to https://cs.stanford.edu/people/matei/ Closes #31512 from pingsutw/SPARK-34158. Authored-by: Kevin Signed-off-by: HyukjinKwon --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 8bb618c..348f9d6 100644 --- a/pom.xml +++ b/pom.xml @@ -48,7 +48,7 @@ matei Matei Zaharia matei.zaha...@gmail.com - http://www.cs.berkeley.edu/~matei + https://cs.stanford.edu/people/matei Apache Software Foundation http://spark.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-34398][DOCS] Fix PySpark migration link
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new de51f25 [SPARK-34398][DOCS] Fix PySpark migration link de51f25 is described below commit de51f25c02894941029ea6e5a029e909465b658a Author: raphaelauv AuthorDate: Mon Feb 8 09:12:15 2021 +0900 [SPARK-34398][DOCS] Fix PySpark migration link ### What changes were proposed in this pull request? docs/pyspark-migration-guide.md ### Why are the changes needed? broken link ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually build and check Closes #31514 from raphaelauv/patch-2. Authored-by: raphaelauv Signed-off-by: HyukjinKwon (cherry picked from commit 34a1a65b398c4469eb97cd458ee172dc76b7ef56) Signed-off-by: HyukjinKwon --- docs/pyspark-migration-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/pyspark-migration-guide.md b/docs/pyspark-migration-guide.md index e117ed1..5a1ca9b 100644 --- a/docs/pyspark-migration-guide.md +++ b/docs/pyspark-migration-guide.md @@ -19,4 +19,4 @@ license: | limitations under the License. --- -The migration guide is now archived on [this page](https://spark.apache.org/docs/latest/api/python/migration_guide/index.html). \ No newline at end of file +The migration guide is now archived on [this page](api/python/migration_guide/index.html). - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (eb5558e -> 34a1a65)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from eb5558e [SPARK-34390][CORE] Enable Zstandard buffer pool by default add 34a1a65 [SPARK-34398][DOCS] Fix PySpark migration link No new revisions were added by this update. Summary of changes: docs/pyspark-migration-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
