This is an automated email from the ASF dual-hosted git repository. gengliang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new e3902d1 [SPARK-35083][FOLLOW-UP][CORE] Add migration guide for the remote scheduler pool files support e3902d1 is described below commit e3902d1975ee6a6a6f672eb6b4f318bcdd237e3f Author: yi.wu <yi...@databricks.com> AuthorDate: Thu Aug 19 16:28:59 2021 +0800 [SPARK-35083][FOLLOW-UP][CORE] Add migration guide for the remote scheduler pool files support ### What changes were proposed in this pull request? Add remote scheduler pool files support to the migration guide. ### Why are the changes needed? To highlight this useful support. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass exiting tests. Closes #33785 from Ngone51/SPARK-35083-follow-up. Lead-authored-by: yi.wu <yi...@databricks.com> Co-authored-by: wuyi <yi...@databricks.com> Signed-off-by: Gengliang Wang <gengli...@apache.org> --- docs/core-migration-guide.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md index 1dee502..02ed430 100644 --- a/docs/core-migration-guide.md +++ b/docs/core-migration-guide.md @@ -24,6 +24,8 @@ license: | ## Upgrading from Core 3.1 to 3.2 +- Since Spark 3.2, the fair scheduler also supports reading a configuration file from a remote node. `spark.scheduler.allocation.file` can either be a local file path or HDFS file path. + - Since Spark 3.2, `spark.hadoopRDD.ignoreEmptySplits` is set to `true` by default which means Spark will not create empty partitions for empty input splits. To restore the behavior before Spark 3.2, you can set `spark.hadoopRDD.ignoreEmptySplits` to `false`. - Since Spark 3.2, `spark.eventLog.compression.codec` is set to `zstd` by default which means Spark will not fallback to use `spark.io.compression.codec` anymore. --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org