[spark] branch master updated (b4a9606 -> 29b7eaa)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b4a9606 [SPARK-31922][CORE] logDebug "RpcEnv already stopped" error on LocalSparkCluster shutdown add 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys No new revisions were added by this update. Summary of changes: .../java/org/apache/hive/service/cli/thrift/ThriftCLIService.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b4a9606 -> 29b7eaa)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b4a9606 [SPARK-31922][CORE] logDebug "RpcEnv already stopped" error on LocalSparkCluster shutdown add 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys No new revisions were added by this update. Summary of changes: .../java/org/apache/hive/service/cli/thrift/ThriftCLIService.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b4a9606 -> 29b7eaa)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b4a9606 [SPARK-31922][CORE] logDebug "RpcEnv already stopped" error on LocalSparkCluster shutdown add 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys No new revisions were added by this update. Summary of changes: .../java/org/apache/hive/service/cli/thrift/ThriftCLIService.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b4a9606 -> 29b7eaa)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b4a9606 [SPARK-31922][CORE] logDebug "RpcEnv already stopped" error on LocalSparkCluster shutdown add 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys No new revisions were added by this update. Summary of changes: .../java/org/apache/hive/service/cli/thrift/ThriftCLIService.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b4a9606 -> 29b7eaa)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b4a9606 [SPARK-31922][CORE] logDebug "RpcEnv already stopped" error on LocalSparkCluster shutdown add 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys No new revisions were added by this update. Summary of changes: .../java/org/apache/hive/service/cli/thrift/ThriftCLIService.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (29b7eaa -> feca9ed)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys add feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests No new revisions were added by this update. Summary of changes: core/src/test/scala/org/apache/spark/rdd/JdbcRDDSuite.scala | 6 -- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala | 6 -- 2 files changed, 8 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (29b7eaa -> feca9ed)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys add feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests No new revisions were added by this update. Summary of changes: core/src/test/scala/org/apache/spark/rdd/JdbcRDDSuite.scala | 6 -- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala | 6 -- 2 files changed, 8 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (29b7eaa -> feca9ed)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys add feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests No new revisions were added by this update. Summary of changes: core/src/test/scala/org/apache/spark/rdd/JdbcRDDSuite.scala | 6 -- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala | 6 -- 2 files changed, 8 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (29b7eaa -> feca9ed)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys add feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests No new revisions were added by this update. Summary of changes: core/src/test/scala/org/apache/spark/rdd/JdbcRDDSuite.scala | 6 -- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala | 6 -- 2 files changed, 8 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (29b7eaa -> feca9ed)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 29b7eaa [MINOR][SQL] Fix warning message for ThriftCLIService.GetCrossReference and GetPrimaryKeys add feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests No new revisions were added by this update. Summary of changes: core/src/test/scala/org/apache/spark/rdd/JdbcRDDSuite.scala | 6 -- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala | 6 -- 2 files changed, 8 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (feca9ed -> 04bf351)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests add 04bf351 [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/mathExpressions.scala | 1 + .../sql-functions/sql-expression-schema.md | 2 +- .../resources/sql-tests/results/operators.sql.out | 24 +++--- .../sql-tests/results/postgreSQL/numeric.sql.out | 22 ++-- 4 files changed, 25 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (feca9ed -> 04bf351)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests add 04bf351 [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/mathExpressions.scala | 1 + .../sql-functions/sql-expression-schema.md | 2 +- .../resources/sql-tests/results/operators.sql.out | 24 +++--- .../sql-tests/results/postgreSQL/numeric.sql.out | 22 ++-- 4 files changed, 25 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (feca9ed -> 04bf351)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests add 04bf351 [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/mathExpressions.scala | 1 + .../sql-functions/sql-expression-schema.md | 2 +- .../resources/sql-tests/results/operators.sql.out | 24 +++--- .../sql-tests/results/postgreSQL/numeric.sql.out | 22 ++-- 4 files changed, 25 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (feca9ed -> 04bf351)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests add 04bf351 [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/mathExpressions.scala | 1 + .../sql-functions/sql-expression-schema.md | 2 +- .../resources/sql-tests/results/operators.sql.out | 24 +++--- .../sql-tests/results/postgreSQL/numeric.sql.out | 22 ++-- 4 files changed, 25 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (feca9ed -> 04bf351)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from feca9ed [MINOR][SQL][TESTS] Create tables once in JDBC tests add 04bf351 [SPARK-21117][SQL][FOLLOWUP] Define prettyName for WidthBucket No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/mathExpressions.scala | 1 + .../sql-functions/sql-expression-schema.md | 2 +- .../resources/sql-tests/results/operators.sql.out | 24 +++--- .../sql-tests/results/postgreSQL/numeric.sql.out | 22 ++-- 4 files changed, 25 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost
This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e8c06af [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost e8c06af is described below commit e8c06af7d1ab19b6478ac542cf876259cac5e13a Author: Wing Yew Poon AuthorDate: Wed Jul 22 09:53:16 2020 -0500 [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost ### What changes were proposed in this pull request? If an executor is lost, the `DAGScheduler` handles the executor loss by removing the executor but does not unregister its outputs if the external shuffle service is used. However, if the node on which the executor runs is lost, the shuffle service may not be able to serve the shuffle files. In such a case, when fetches from the executor's outputs fail in the same stage, the `DAGScheduler` again removes the executor and by right, should unregister its outputs. It doesn't because the epoch used to track the executor failure has not increased. We track the epoch for failed executors that result in lost file output separately, so we can unregister the outputs in this scenario. The idea to track a second epoch is due to Attila Zsolt Piros. ### Why are the changes needed? Without the changes, the loss of a node could require two stage attempts to recover instead of one. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New unit test. This test fails without the change and passes with it. Closes #28848 from wypoon/SPARK-32003. Authored-by: Wing Yew Poon Signed-off-by: Imran Rashid --- .../org/apache/spark/scheduler/DAGScheduler.scala | 100 ++--- .../apache/spark/scheduler/DAGSchedulerSuite.scala | 98 2 files changed, 148 insertions(+), 50 deletions(-) diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala index cb024d0..73c95d1 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala @@ -170,13 +170,34 @@ private[spark] class DAGScheduler( */ private val cacheLocs = new HashMap[Int, IndexedSeq[Seq[TaskLocation]]] - // For tracking failed nodes, we use the MapOutputTracker's epoch number, which is sent with - // every task. When we detect a node failing, we note the current epoch number and failed - // executor, increment it for new tasks, and use this to ignore stray ShuffleMapTask results. - // - // TODO: Garbage collect information about failure epochs when we know there are no more - // stray messages to detect. - private val failedEpoch = new HashMap[String, Long] + /** + * Tracks the latest epoch of a fully processed error related to the given executor. (We use + * the MapOutputTracker's epoch number, which is sent with every task.) + * + * When an executor fails, it can affect the results of many tasks, and we have to deal with + * all of them consistently. We don't simply ignore all future results from that executor, + * as the failures may have been transient; but we also don't want to "overreact" to follow- + * on errors we receive. Furthermore, we might receive notification of a task success, after + * we find out the executor has actually failed; we'll assume those successes are, in fact, + * simply delayed notifications and the results have been lost, if the tasks started in the + * same or an earlier epoch. In particular, we use this to control when we tell the + * BlockManagerMaster that the BlockManager has been lost. + */ + private val executorFailureEpoch = new HashMap[String, Long] + + /** + * Tracks the latest epoch of a fully processed error where shuffle files have been lost from + * the given executor. + * + * This is closely related to executorFailureEpoch. They only differ for the executor when + * there is an external shuffle service serving shuffle files and we haven't been notified that + * the entire worker has been lost. In that case, when an executor is lost, we do not update + * the shuffleFileLostEpoch; we wait for a fetch failure. This way, if only the executor + * fails, we do not unregister the shuffle data as it can still be served; but if there is + * a failure in the shuffle service (resulting in fetch failure), we unregister the shuffle + * data only once, even if we get many fetch failures. + */ + private val shuffleFileLostEpoch = new HashMap[String, Long] private [scheduler] val outputComm
[spark] branch master updated (e8c06af -> cd16a10)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e8c06af [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost add cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e8c06af -> cd16a10)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e8c06af [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost add cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e8c06af -> cd16a10)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e8c06af [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost add cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (9729144 -> ad9e7a2)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 9729144 [SPARK-32377][SQL] CaseInsensitiveMap should be deterministic for addition add ad9e7a2 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (9729144 -> ad9e7a2)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 9729144 [SPARK-32377][SQL] CaseInsensitiveMap should be deterministic for addition add ad9e7a2 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (9729144 -> ad9e7a2)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 9729144 [SPARK-32377][SQL] CaseInsensitiveMap should be deterministic for addition add ad9e7a2 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (9729144 -> ad9e7a2)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 9729144 [SPARK-32377][SQL] CaseInsensitiveMap should be deterministic for addition add ad9e7a2 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e8c06af -> cd16a10)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e8c06af [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost add cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (9729144 -> ad9e7a2)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 9729144 [SPARK-32377][SQL] CaseInsensitiveMap should be deterministic for addition add ad9e7a2 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e8c06af -> cd16a10)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e8c06af [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost add cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../sql/catalyst/util/CaseInsensitiveMap.scala | 6 +- .../org/apache/spark/sql/DataFrameReader.scala | 10 +- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +++-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 3 ++- .../sql/test/DataFrameReaderWriterSuite.scala | 22 ++ 6 files changed, 42 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cd16a10 -> 184074d)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for
DataFrameReader/Writer options
add 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command
No new revisions were added by this update.
Summary of changes:
docs/_data/menu-sql.yaml | 2 +
docs/sql-ref-syntax-aux-cache-cache-table.md | 1 +
docs/sql-ref-syntax-aux-cache-clear-cache.md | 1 +
...> sql-ref-syntax-aux-cache-refresh-function.md} | 32 ---
docs/sql-ref-syntax-aux-cache-refresh-table.md | 1 +
docs/sql-ref-syntax-aux-cache-refresh.md | 1 +
docs/sql-ref-syntax-aux-cache-uncache-table.md | 1 +
docs/sql-ref-syntax-aux-cache.md | 3 +-
docs/sql-ref-syntax.md | 1 +
.../apache/spark/sql/catalyst/parser/SqlBase.g4| 1 +
.../spark/sql/catalyst/analysis/Analyzer.scala | 6 +++
.../sql/catalyst/analysis/v2ResolutionPlans.scala | 20 +
.../sql/catalyst/catalog/SessionCatalog.scala | 8
.../spark/sql/catalyst/parser/AstBuilder.scala | 5 +++
.../sql/catalyst/plans/logical/v2Commands.scala| 7
.../sql/connector/catalog/CatalogV2Implicits.scala | 10 -
.../sql/connector/catalog/LookupCatalog.scala | 29 -
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 11 -
.../catalyst/analysis/ResolveSessionCatalog.scala | 30 ++
.../spark/sql/execution/command/functions.scala| 39 ++
.../spark/sql/connector/DataSourceV2SQLSuite.scala | 13 ++
.../spark/sql/execution/command/DDLSuite.scala | 47 +-
22 files changed, 223 insertions(+), 46 deletions(-)
copy docs/{sql-ref-syntax-aux-cache-refresh.md =>
sql-ref-syntax-aux-cache-refresh-function.md} (53%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cd16a10 -> 184074d)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for
DataFrameReader/Writer options
add 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command
No new revisions were added by this update.
Summary of changes:
docs/_data/menu-sql.yaml | 2 +
docs/sql-ref-syntax-aux-cache-cache-table.md | 1 +
docs/sql-ref-syntax-aux-cache-clear-cache.md | 1 +
...> sql-ref-syntax-aux-cache-refresh-function.md} | 32 ---
docs/sql-ref-syntax-aux-cache-refresh-table.md | 1 +
docs/sql-ref-syntax-aux-cache-refresh.md | 1 +
docs/sql-ref-syntax-aux-cache-uncache-table.md | 1 +
docs/sql-ref-syntax-aux-cache.md | 3 +-
docs/sql-ref-syntax.md | 1 +
.../apache/spark/sql/catalyst/parser/SqlBase.g4| 1 +
.../spark/sql/catalyst/analysis/Analyzer.scala | 6 +++
.../sql/catalyst/analysis/v2ResolutionPlans.scala | 20 +
.../sql/catalyst/catalog/SessionCatalog.scala | 8
.../spark/sql/catalyst/parser/AstBuilder.scala | 5 +++
.../sql/catalyst/plans/logical/v2Commands.scala| 7
.../sql/connector/catalog/CatalogV2Implicits.scala | 10 -
.../sql/connector/catalog/LookupCatalog.scala | 29 -
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 11 -
.../catalyst/analysis/ResolveSessionCatalog.scala | 30 ++
.../spark/sql/execution/command/functions.scala| 39 ++
.../spark/sql/connector/DataSourceV2SQLSuite.scala | 13 ++
.../spark/sql/execution/command/DDLSuite.scala | 47 +-
22 files changed, 223 insertions(+), 46 deletions(-)
copy docs/{sql-ref-syntax-aux-cache-refresh.md =>
sql-ref-syntax-aux-cache-refresh-function.md} (53%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cd16a10 -> 184074d)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for
DataFrameReader/Writer options
add 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command
No new revisions were added by this update.
Summary of changes:
docs/_data/menu-sql.yaml | 2 +
docs/sql-ref-syntax-aux-cache-cache-table.md | 1 +
docs/sql-ref-syntax-aux-cache-clear-cache.md | 1 +
...> sql-ref-syntax-aux-cache-refresh-function.md} | 32 ---
docs/sql-ref-syntax-aux-cache-refresh-table.md | 1 +
docs/sql-ref-syntax-aux-cache-refresh.md | 1 +
docs/sql-ref-syntax-aux-cache-uncache-table.md | 1 +
docs/sql-ref-syntax-aux-cache.md | 3 +-
docs/sql-ref-syntax.md | 1 +
.../apache/spark/sql/catalyst/parser/SqlBase.g4| 1 +
.../spark/sql/catalyst/analysis/Analyzer.scala | 6 +++
.../sql/catalyst/analysis/v2ResolutionPlans.scala | 20 +
.../sql/catalyst/catalog/SessionCatalog.scala | 8
.../spark/sql/catalyst/parser/AstBuilder.scala | 5 +++
.../sql/catalyst/plans/logical/v2Commands.scala| 7
.../sql/connector/catalog/CatalogV2Implicits.scala | 10 -
.../sql/connector/catalog/LookupCatalog.scala | 29 -
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 11 -
.../catalyst/analysis/ResolveSessionCatalog.scala | 30 ++
.../spark/sql/execution/command/functions.scala| 39 ++
.../spark/sql/connector/DataSourceV2SQLSuite.scala | 13 ++
.../spark/sql/execution/command/DDLSuite.scala | 47 +-
22 files changed, 223 insertions(+), 46 deletions(-)
copy docs/{sql-ref-syntax-aux-cache-refresh.md =>
sql-ref-syntax-aux-cache-refresh-function.md} (53%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cd16a10 -> 184074d)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for
DataFrameReader/Writer options
add 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command
No new revisions were added by this update.
Summary of changes:
docs/_data/menu-sql.yaml | 2 +
docs/sql-ref-syntax-aux-cache-cache-table.md | 1 +
docs/sql-ref-syntax-aux-cache-clear-cache.md | 1 +
...> sql-ref-syntax-aux-cache-refresh-function.md} | 32 ---
docs/sql-ref-syntax-aux-cache-refresh-table.md | 1 +
docs/sql-ref-syntax-aux-cache-refresh.md | 1 +
docs/sql-ref-syntax-aux-cache-uncache-table.md | 1 +
docs/sql-ref-syntax-aux-cache.md | 3 +-
docs/sql-ref-syntax.md | 1 +
.../apache/spark/sql/catalyst/parser/SqlBase.g4| 1 +
.../spark/sql/catalyst/analysis/Analyzer.scala | 6 +++
.../sql/catalyst/analysis/v2ResolutionPlans.scala | 20 +
.../sql/catalyst/catalog/SessionCatalog.scala | 8
.../spark/sql/catalyst/parser/AstBuilder.scala | 5 +++
.../sql/catalyst/plans/logical/v2Commands.scala| 7
.../sql/connector/catalog/CatalogV2Implicits.scala | 10 -
.../sql/connector/catalog/LookupCatalog.scala | 29 -
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 11 -
.../catalyst/analysis/ResolveSessionCatalog.scala | 30 ++
.../spark/sql/execution/command/functions.scala| 39 ++
.../spark/sql/connector/DataSourceV2SQLSuite.scala | 13 ++
.../spark/sql/execution/command/DDLSuite.scala | 47 +-
22 files changed, 223 insertions(+), 46 deletions(-)
copy docs/{sql-ref-syntax-aux-cache-refresh.md =>
sql-ref-syntax-aux-cache-refresh-function.md} (53%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cd16a10 -> 184074d)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from cd16a10 [SPARK-32364][SQL] Use CaseInsensitiveMap for
DataFrameReader/Writer options
add 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command
No new revisions were added by this update.
Summary of changes:
docs/_data/menu-sql.yaml | 2 +
docs/sql-ref-syntax-aux-cache-cache-table.md | 1 +
docs/sql-ref-syntax-aux-cache-clear-cache.md | 1 +
...> sql-ref-syntax-aux-cache-refresh-function.md} | 32 ---
docs/sql-ref-syntax-aux-cache-refresh-table.md | 1 +
docs/sql-ref-syntax-aux-cache-refresh.md | 1 +
docs/sql-ref-syntax-aux-cache-uncache-table.md | 1 +
docs/sql-ref-syntax-aux-cache.md | 3 +-
docs/sql-ref-syntax.md | 1 +
.../apache/spark/sql/catalyst/parser/SqlBase.g4| 1 +
.../spark/sql/catalyst/analysis/Analyzer.scala | 6 +++
.../sql/catalyst/analysis/v2ResolutionPlans.scala | 20 +
.../sql/catalyst/catalog/SessionCatalog.scala | 8
.../spark/sql/catalyst/parser/AstBuilder.scala | 5 +++
.../sql/catalyst/plans/logical/v2Commands.scala| 7
.../sql/connector/catalog/CatalogV2Implicits.scala | 10 -
.../sql/connector/catalog/LookupCatalog.scala | 29 -
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 11 -
.../catalyst/analysis/ResolveSessionCatalog.scala | 30 ++
.../spark/sql/execution/command/functions.scala| 39 ++
.../spark/sql/connector/DataSourceV2SQLSuite.scala | 13 ++
.../spark/sql/execution/command/DDLSuite.scala | 47 +-
22 files changed, 223 insertions(+), 46 deletions(-)
copy docs/{sql-ref-syntax-aux-cache-refresh.md =>
sql-ref-syntax-aux-cache-refresh-function.md} (53%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (184074d -> b151194)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command add b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server No new revisions were added by this update. Summary of changes: .../spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala | 4 +--- sql/hive-thriftserver/src/test/resources/log4j.properties | 2 ++ 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (184074d -> b151194)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command add b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server No new revisions were added by this update. Summary of changes: .../spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala | 4 +--- sql/hive-thriftserver/src/test/resources/log4j.properties | 2 ++ 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (184074d -> b151194)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command add b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server No new revisions were added by this update. Summary of changes: .../spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala | 4 +--- sql/hive-thriftserver/src/test/resources/log4j.properties | 2 ++ 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (184074d -> b151194)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command add b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server No new revisions were added by this update. Summary of changes: .../spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala | 4 +--- sql/hive-thriftserver/src/test/resources/log4j.properties | 2 ++ 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (184074d -> b151194)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 184074d [SPARK-31999][SQL] Add REFRESH FUNCTION command add b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server No new revisions were added by this update. Summary of changes: .../spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala | 4 +--- sql/hive-thriftserver/src/test/resources/log4j.properties | 2 ++ 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b151194 -> 4616982)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server add 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala| 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b151194 -> 4616982)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server add 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala| 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b151194 -> 4616982)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server add 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala| 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b151194 -> 4616982)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server add 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala| 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4616982 -> 182566b)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc add 182566b [SPARK-32237][SQL] Resolve hint in CTE No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 10 - .../sql/catalyst/analysis/AnalysisSuite.scala | 25 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 12 +++ 3 files changed, 41 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b151194 -> 4616982)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b151194 [SPARK-32392][SQL] Reduce duplicate error log for executing sql statement operation in thrift server add 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala| 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4616982 -> 182566b)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc add 182566b [SPARK-32237][SQL] Resolve hint in CTE No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 10 - .../sql/catalyst/analysis/AnalysisSuite.scala | 25 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 12 +++ 3 files changed, 41 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4616982 -> 182566b)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc add 182566b [SPARK-32237][SQL] Resolve hint in CTE No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 10 - .../sql/catalyst/analysis/AnalysisSuite.scala | 25 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 12 +++ 3 files changed, 41 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4616982 -> 182566b)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc add 182566b [SPARK-32237][SQL] Resolve hint in CTE No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 10 - .../sql/catalyst/analysis/AnalysisSuite.scala | 25 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 12 +++ 3 files changed, 41 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4616982 -> 182566b)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4616982 [SPARK-30616][SQL][FOLLOW-UP] Use only config key name in the config doc add 182566b [SPARK-32237][SQL] Resolve hint in CTE No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 10 - .../sql/catalyst/analysis/AnalysisSuite.scala | 25 +- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 12 +++ 3 files changed, 41 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor
This is an automated email from the ASF dual-hosted git repository.
holden pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f8d29d3 [SPARK-32217] Plumb whether a worker would also be
decommissioned along with executor
f8d29d3 is described below
commit f8d29d371cdf0b8be6a48a9124ffbc3c0794f32a
Author: Devesh Agrawal
AuthorDate: Wed Jul 22 21:04:06 2020 -0700
[SPARK-32217] Plumb whether a worker would also be decommissioned along
with executor
### What changes were proposed in this pull request?
This PR is a giant plumbing PR that plumbs an `ExecutorDecommissionInfo`
along
with the DecommissionExecutor message.
### Why are the changes needed?
The primary motivation is to know whether a decommissioned executor
would also be loosing shuffle files -- and thus it is important to know
whether the host would also be decommissioned.
In the absence of this PR, the existing code assumes that decommissioning
an executor does not loose the whole host with it, and thus does not clear the
shuffle state if external shuffle service is enabled. While this may hold in
some cases (like K8s decommissioning an executor pod, or YARN container
preemption), it does not hold in others like when the cluster is managed by a
Standalone Scheduler (Master). This is similar to the existing `workerLost`
field in the `ExecutorProcessLost [...]
In the future, this `ExecutorDecommissionInfo` can be embellished for
knowing how long the executor has to live for scenarios like Cloud spot
kills (or Yarn preemption) and the like.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tweaked an existing unit test in `AppClientSuite`
Closes #29032 from agrawaldevesh/plumb_decom_info.
Authored-by: Devesh Agrawal
Signed-off-by: Holden Karau
---
.../spark/deploy/client/StandaloneAppClient.scala | 4 +++-
.../client/StandaloneAppClientListener.scala | 4 +++-
.../org/apache/spark/deploy/master/Master.scala| 5 +++-
.../executor/CoarseGrainedExecutorBackend.scala| 16 +
.../spark/scheduler/ExecutorDecommissionInfo.scala | 28 ++
.../org/apache/spark/scheduler/TaskScheduler.scala | 2 +-
.../apache/spark/scheduler/TaskSchedulerImpl.scala | 3 ++-
.../cluster/CoarseGrainedClusterMessage.scala | 4 +++-
.../cluster/CoarseGrainedSchedulerBackend.scala| 22 +
.../cluster/StandaloneSchedulerBackend.scala | 6 ++---
.../spark/deploy/client/AppClientSuite.scala | 15
.../apache/spark/scheduler/DAGSchedulerSuite.scala | 8 +--
.../scheduler/ExternalClusterManagerSuite.scala| 4 +++-
.../WorkerDecommissionExtendedSuite.scala | 2 +-
.../spark/scheduler/WorkerDecommissionSuite.scala | 2 +-
.../BlockManagerDecommissionIntegrationSuite.scala | 2 +-
16 files changed, 92 insertions(+), 35 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClient.scala
b/core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClient.scala
index eedf5e9..a6da839 100644
---
a/core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClient.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClient.scala
@@ -31,6 +31,7 @@ import org.apache.spark.deploy.DeployMessages._
import org.apache.spark.deploy.master.Master
import org.apache.spark.internal.Logging
import org.apache.spark.rpc._
+import org.apache.spark.scheduler.ExecutorDecommissionInfo
import org.apache.spark.util.{RpcUtils, ThreadUtils}
/**
@@ -181,7 +182,8 @@ private[spark] class StandaloneAppClient(
if (ExecutorState.isFinished(state)) {
listener.executorRemoved(fullId, message.getOrElse(""), exitStatus,
workerLost)
} else if (state == ExecutorState.DECOMMISSIONED) {
- listener.executorDecommissioned(fullId, message.getOrElse(""))
+ listener.executorDecommissioned(fullId,
+ExecutorDecommissionInfo(message.getOrElse(""),
isHostDecommissioned = workerLost))
}
case WorkerRemoved(id, host, message) =>
diff --git
a/core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClientListener.scala
b/core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClientListener.scala
index 2e38a68..e72f7e9 100644
---
a/core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClientListener.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClientListener.scala
@@ -17,6 +17,8 @@
package org.apache.spark.deploy.client
+import org.apache.spark.scheduler.ExecutorDecommissionInfo
+
/**
* Callbacks invoked by deploy client when various events happen. There are
currently five events:
* connecting to the clust
[spark] branch master updated (f8d29d3 -> 7b66882)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f8d29d3 [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor add 7b66882 [SPARK-32338][SQL][PYSPARK][FOLLOW-UP] Update slice to accept Column for start and length No new revisions were added by this update. Summary of changes: python/pyspark/sql/functions.py| 6 +- python/pyspark/sql/tests/test_functions.py | 10 ++ 2 files changed, 15 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f8d29d3 -> 7b66882)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f8d29d3 [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor add 7b66882 [SPARK-32338][SQL][PYSPARK][FOLLOW-UP] Update slice to accept Column for start and length No new revisions were added by this update. Summary of changes: python/pyspark/sql/functions.py| 6 +- python/pyspark/sql/tests/test_functions.py | 10 ++ 2 files changed, 15 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f8d29d3 -> 7b66882)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f8d29d3 [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor add 7b66882 [SPARK-32338][SQL][PYSPARK][FOLLOW-UP] Update slice to accept Column for start and length No new revisions were added by this update. Summary of changes: python/pyspark/sql/functions.py| 6 +- python/pyspark/sql/tests/test_functions.py | 10 ++ 2 files changed, 15 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f8d29d3 -> 7b66882)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f8d29d3 [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor add 7b66882 [SPARK-32338][SQL][PYSPARK][FOLLOW-UP] Update slice to accept Column for start and length No new revisions were added by this update. Summary of changes: python/pyspark/sql/functions.py| 6 +- python/pyspark/sql/tests/test_functions.py | 10 ++ 2 files changed, 15 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f8d29d3 -> 7b66882)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f8d29d3 [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor add 7b66882 [SPARK-32338][SQL][PYSPARK][FOLLOW-UP] Update slice to accept Column for start and length No new revisions were added by this update. Summary of changes: python/pyspark/sql/functions.py| 6 +- python/pyspark/sql/tests/test_functions.py | 10 ++ 2 files changed, 15 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
