[spark] branch master updated (891c5e6 -> e15ae60)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 891c5e6 [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation add e15ae60 [SPARK-32550][SQL] Make SpecificInternalRow constructors faster by using while loops instead of maps No new revisions were added by this update. Summary of changes: .../catalyst/expressions/SpecificInternalRow.scala | 49 +++--- 1 file changed, 34 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f33b64a -> 891c5e6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder add 891c5e6 [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation No new revisions were added by this update. Summary of changes: python/docs/source/reference/pyspark.ml.rst | 1 + python/docs/source/reference/pyspark.rst| 1 + python/pyspark/ml/clustering.py | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f33b64a -> 891c5e6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder add 891c5e6 [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation No new revisions were added by this update. Summary of changes: python/docs/source/reference/pyspark.ml.rst | 1 + python/docs/source/reference/pyspark.rst| 1 + python/pyspark/ml/clustering.py | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f33b64a -> 891c5e6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder add 891c5e6 [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation No new revisions were added by this update. Summary of changes: python/docs/source/reference/pyspark.ml.rst | 1 + python/docs/source/reference/pyspark.rst| 1 + python/pyspark/ml/clustering.py | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f33b64a -> 891c5e6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder add 891c5e6 [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation No new revisions were added by this update. Summary of changes: python/docs/source/reference/pyspark.ml.rst | 1 + python/docs/source/reference/pyspark.rst| 1 + python/pyspark/ml/clustering.py | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f33b64a -> 891c5e6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder add 891c5e6 [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation No new revisions were added by this update. Summary of changes: python/docs/source/reference/pyspark.ml.rst | 1 + python/docs/source/reference/pyspark.rst| 1 + python/pyspark/ml/clustering.py | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (a36514e -> 753d414)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from a36514e [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources add 753d414 [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (a36514e -> 753d414)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from a36514e [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources add 753d414 [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1b39215 -> f33b64a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation add f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (a36514e -> 753d414)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from a36514e [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources add 753d414 [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1b39215 -> f33b64a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation add f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (a36514e -> 753d414)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from a36514e [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources add 753d414 [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1b39215 -> f33b64a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation add f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (a36514e -> 753d414)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from a36514e [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources add 753d414 [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1b39215 -> f33b64a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation add f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1b39215 -> f33b64a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation add f33b64a [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/objects.scala | 10 ++- .../optimizer/ObjectSerializerPruningSuite.scala | 33 +- .../spark/sql/DatasetOptimizationSuite.scala | 13 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (70964e7 -> bfd8c34)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled add bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact No new revisions were added by this update. Summary of changes: .github/workflows/build_and_test.yml | 6 ++ 1 file changed, 6 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfd8c34 -> 1b39215)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact add 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 2 ++ 1 file changed, 2 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfd8c34 -> 1b39215)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact add 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 2 ++ 1 file changed, 2 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfd8c34 -> 1b39215)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact add 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 2 ++ 1 file changed, 2 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (70964e7 -> bfd8c34)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled add bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact No new revisions were added by this update. Summary of changes: .github/workflows/build_and_test.yml | 6 ++ 1 file changed, 6 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfd8c34 -> 1b39215)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact add 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 2 ++ 1 file changed, 2 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (70964e7 -> bfd8c34)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled add bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact No new revisions were added by this update. Summary of changes: .github/workflows/build_and_test.yml | 6 ++ 1 file changed, 6 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfd8c34 -> 1b39215)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact add 1b39215 [SPARK-32018][FOLLOWUP][DOC] Add migration guide for decimal value overflow in sum aggregation No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 2 ++ 1 file changed, 2 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (70964e7 -> bfd8c34)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled add bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact No new revisions were added by this update. Summary of changes: .github/workflows/build_and_test.yml | 6 ++ 1 file changed, 6 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (70964e7 -> bfd8c34)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled add bfd8c34 [SPARK-32645][INFRA] Upload unit-tests.log as an artifact No new revisions were added by this update. Summary of changes: .github/workflows/build_and_test.yml | 6 ++ 1 file changed, 6 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (babb654 -> 70964e7)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter add 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/scheduler/TaskSetManager.scala | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (babb654 -> 70964e7)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter add 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/scheduler/TaskSetManager.scala | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (babb654 -> 70964e7)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter add 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/scheduler/TaskSetManager.scala | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (babb654 -> 70964e7)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter add 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/scheduler/TaskSetManager.scala | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (babb654 -> 70964e7)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter add 70964e7 [SPARK-21040][CORE][FOLLOW-UP] Only calculate executorKillTime when speculation is enabled No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/scheduler/TaskSetManager.scala | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d0dfe49 -> babb654)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml add babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter No new revisions were added by this update. Summary of changes: R/pkg/tests/run-all.R | 9 ++--- appveyor.yml | 4 ++-- 2 files changed, 8 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d0dfe49 -> babb654)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml add babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter No new revisions were added by this update. Summary of changes: R/pkg/tests/run-all.R | 9 ++--- appveyor.yml | 4 ++-- 2 files changed, 8 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d0dfe49 -> babb654)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml add babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter No new revisions were added by this update. Summary of changes: R/pkg/tests/run-all.R | 9 ++--- appveyor.yml | 4 ++-- 2 files changed, 8 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d0dfe49 -> babb654)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml add babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter No new revisions were added by this update. Summary of changes: R/pkg/tests/run-all.R | 9 ++--- appveyor.yml | 4 ++-- 2 files changed, 8 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d0dfe49 -> babb654)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml add babb654 [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter No new revisions were added by this update. Summary of changes: R/pkg/tests/run-all.R | 9 ++--- appveyor.yml | 4 ++-- 2 files changed, 8 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (21e0dd04 -> d0dfe49)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 21e0dd04 [SPARK-32119][FOLLOWUP][DOC] Update monitoring doc following
the improvement in SPARK-32119
add d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml
No new revisions were added by this update.
Summary of changes:
.github/workflows/{master.yml => build_and_test.yml} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename .github/workflows/{master.yml => build_and_test.yml} (100%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (21e0dd04 -> d0dfe49)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 21e0dd04 [SPARK-32119][FOLLOWUP][DOC] Update monitoring doc following
the improvement in SPARK-32119
add d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml
No new revisions were added by this update.
Summary of changes:
.github/workflows/{master.yml => build_and_test.yml} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename .github/workflows/{master.yml => build_and_test.yml} (100%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (21e0dd04 -> d0dfe49)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 21e0dd04 [SPARK-32119][FOLLOWUP][DOC] Update monitoring doc following
the improvement in SPARK-32119
add d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml
No new revisions were added by this update.
Summary of changes:
.github/workflows/{master.yml => build_and_test.yml} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename .github/workflows/{master.yml => build_and_test.yml} (100%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (21e0dd04 -> d0dfe49)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 21e0dd04 [SPARK-32119][FOLLOWUP][DOC] Update monitoring doc following
the improvement in SPARK-32119
add d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml
No new revisions were added by this update.
Summary of changes:
.github/workflows/{master.yml => build_and_test.yml} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename .github/workflows/{master.yml => build_and_test.yml} (100%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (21e0dd04 -> d0dfe49)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 21e0dd04 [SPARK-32119][FOLLOWUP][DOC] Update monitoring doc following
the improvement in SPARK-32119
add d0dfe49 [MINOR][INFRA] Rename master.yml to build_and_test.yml
No new revisions were added by this update.
Summary of changes:
.github/workflows/{master.yml => build_and_test.yml} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename .github/workflows/{master.yml => build_and_test.yml} (100%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-32119][FOLLOWUP][DOC] Update monitoring doc following the improvement in SPARK-32119
This is an automated email from the ASF dual-hosted git repository. sarutak pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 21e0dd04 [SPARK-32119][FOLLOWUP][DOC] Update monitoring doc following the improvement in SPARK-32119 21e0dd04 is described below commit 21e0dd0461ef6496bea1436687fae46463f310b2 Author: Luca Canali AuthorDate: Tue Aug 18 18:53:34 2020 +0900 [SPARK-32119][FOLLOWUP][DOC] Update monitoring doc following the improvement in SPARK-32119 ### What changes were proposed in this pull request? Update monitoring doc following the improvement/fix in SPARK-32119. ### Why are the changes needed? SPARK-32119 removes the limitations listed in the monitoring doc "Distribution of the jar files containing the plugin code is currently not done by Spark." ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Not relevant Closes #29463 from LucaCanali/followupSPARK32119. Authored-by: Luca Canali Signed-off-by: Kousuke Saruta --- docs/monitoring.md | 6 -- 1 file changed, 6 deletions(-) diff --git a/docs/monitoring.md b/docs/monitoring.md index 31fc160..97948f6 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -1366,9 +1366,3 @@ Both take a comma-separated list of class names that implement the possible for one list to be placed in the Spark default config file, allowing users to easily add other plugins from the command line without overwriting the config file's list. Duplicate plugins are ignored. - -Distribution of the jar files containing the plugin code is currently not done by Spark. The user -or admin should make sure that the jar files are available to Spark applications, for example, by -including the plugin jar with the Spark distribution. The exception to this rule is the YARN -backend, where the --jars command line option (or equivalent config entry) can be -used to make the plugin code available to both executors and cluster-mode drivers. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (6cdc32f -> a36514e)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 6cdc32f [SPARK-32622][SQL][TEST] Add case-sensitivity test for ORC predicate pushdown add a36514e [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/SparkContext.scala | 10 +++--- .../org/apache/spark/internal/config/Tests.scala | 15 + .../scheduler/BarrierJobAllocationFailed.scala | 4 +-- .../org/apache/spark/scheduler/DAGScheduler.scala | 10 +++--- .../spark/scheduler/ExecutorResourceInfo.scala | 1 + .../apache/spark/scheduler/SchedulerBackend.scala | 3 +- .../apache/spark/scheduler/TaskSchedulerImpl.scala | 36 +- .../cluster/CoarseGrainedSchedulerBackend.scala| 22 +++-- .../spark/BarrierStageOnSubmittedSuite.scala | 36 ++ .../scala/org/apache/spark/SparkContextSuite.scala | 2 ++ 10 files changed, 124 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (6cdc32f -> a36514e)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 6cdc32f [SPARK-32622][SQL][TEST] Add case-sensitivity test for ORC predicate pushdown add a36514e [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/SparkContext.scala | 10 +++--- .../org/apache/spark/internal/config/Tests.scala | 15 + .../scheduler/BarrierJobAllocationFailed.scala | 4 +-- .../org/apache/spark/scheduler/DAGScheduler.scala | 10 +++--- .../spark/scheduler/ExecutorResourceInfo.scala | 1 + .../apache/spark/scheduler/SchedulerBackend.scala | 3 +- .../apache/spark/scheduler/TaskSchedulerImpl.scala | 36 +- .../cluster/CoarseGrainedSchedulerBackend.scala| 22 +++-- .../spark/BarrierStageOnSubmittedSuite.scala | 36 ++ .../scala/org/apache/spark/SparkContextSuite.scala | 2 ++ 10 files changed, 124 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b33066f -> 1ac23de)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b33066f [SPARK-32622][SQL][TEST] Add case-sensitivity test for ORC predicate pushdown add 1ac23de [SPARK-32613][CORE] Fix regressions in DecommissionWorkerSuite No new revisions were added by this update. Summary of changes: .../executor/CoarseGrainedExecutorBackend.scala| 10 +++- .../org/apache/spark/internal/config/package.scala | 10 .../org/apache/spark/scheduler/DAGScheduler.scala | 41 ++ .../apache/spark/scheduler/TaskSchedulerImpl.scala | 29 -- .../spark/deploy/DecommissionWorkerSuite.scala | 64 ++ .../spark/scheduler/TaskSchedulerImplSuite.scala | 36 +--- 6 files changed, 153 insertions(+), 37 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a36514e [3.0][SPARK-32518][CORE]
CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds
of resources
a36514e is described below
commit a36514e1fbf6920f0d18a7d8fb690822843eb2a2
Author: yi.wu
AuthorDate: Tue Aug 18 06:50:05 2020 +
[3.0][SPARK-32518][CORE]
CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds
of resources
### What changes were proposed in this pull request?
1. Make `CoarseGrainedSchedulerBackend.maxNumConcurrentTasks()` considers
all kinds of resources when calculating the max concurrent tasks
2. Refactor `calculateAvailableSlots()` to make it be able to be used for
both `CoarseGrainedSchedulerBackend` and `TaskSchedulerImpl`
### Why are the changes needed?
Currently, `CoarseGrainedSchedulerBackend.maxNumConcurrentTasks()` only
considers the CPU for the max concurrent tasks. This can cause the application
to hang when a barrier stage requires extra custom resources but the cluster
doesn't have enough corresponding resources. Because, without the checking for
other custom resources in `maxNumConcurrentTasks`, the barrier stage can be
submitted to the `TaskSchedulerImpl`. But the `TaskSchedulerImpl` won't launch
tasks for the barrier stage [...]
If the barrier stage doesn't launch all the tasks in one true, the
application will fail and suggest user to disable delay scheduling. However,
this actually a misleading suggestion since the real root cause is not enough
resources.
### Does this PR introduce _any_ user-facing change?
Yes. In case of a barrier stage requires more custom resources than the
cluster has, previously, the application will fail with misleading suggestion
of disabling delay scheduling. After this PR, the application will fail with
the error message saying not enough resources.
### How was this patch tested?
Added a unit test.
Closes #29395 from Ngone51/backport-spark-32518.
Authored-by: yi.wu
Signed-off-by: Wenchen Fan
---
.../main/scala/org/apache/spark/SparkContext.scala | 10 +++---
.../org/apache/spark/internal/config/Tests.scala | 15 +
.../scheduler/BarrierJobAllocationFailed.scala | 4 +--
.../org/apache/spark/scheduler/DAGScheduler.scala | 10 +++---
.../spark/scheduler/ExecutorResourceInfo.scala | 1 +
.../apache/spark/scheduler/SchedulerBackend.scala | 3 +-
.../apache/spark/scheduler/TaskSchedulerImpl.scala | 36 +-
.../cluster/CoarseGrainedSchedulerBackend.scala| 22 +++--
.../spark/BarrierStageOnSubmittedSuite.scala | 36 ++
.../scala/org/apache/spark/SparkContextSuite.scala | 2 ++
10 files changed, 124 insertions(+), 15 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 65c08cf..66fe1d7 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -1597,7 +1597,8 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get the max number of tasks that can be concurrent launched currently.
+ * Get the max number of tasks that can be concurrent launched based on the
resources
+ * could be used, even if some of them are being used at the moment.
* Note that please don't cache the value returned by this method, because
the number can change
* due to add/remove executors.
*
@@ -2776,8 +2777,9 @@ object SparkContext extends Logging {
}
// some cluster managers don't set the EXECUTOR_CORES config by default
(standalone
// and mesos coarse grained), so we can't rely on that config for those.
- val shouldCheckExecCores = executorCores.isDefined ||
sc.conf.contains(EXECUTOR_CORES) ||
+ var shouldCheckExecCores = executorCores.isDefined ||
sc.conf.contains(EXECUTOR_CORES) ||
(master.equalsIgnoreCase("yarn") || master.startsWith("k8s"))
+ shouldCheckExecCores &= !sc.conf.get(SKIP_VALIDATE_CORES_TESTING)
// Number of cores per executor must meet at least one task requirement.
if (shouldCheckExecCores && execCores < taskCores) {
@@ -2833,7 +2835,7 @@ object SparkContext extends Logging {
limitingResourceName = taskReq.resourceName
}
}
- if(!shouldCheckExecCores && Utils.isDynamicAllocationEnabled(sc.conf)) {
+ if(!shouldCheckExecCores) {
// if we can't rely on the executor cores config throw a warning for
user
logWarning("Please ensure that the number of slots available on your "
+
"executors is limited by the number
[spark] branch master updated (b33066f -> 1ac23de)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b33066f [SPARK-32622][SQL][TEST] Add case-sensitivity test for ORC predicate pushdown add 1ac23de [SPARK-32613][CORE] Fix regressions in DecommissionWorkerSuite No new revisions were added by this update. Summary of changes: .../executor/CoarseGrainedExecutorBackend.scala| 10 +++- .../org/apache/spark/internal/config/package.scala | 10 .../org/apache/spark/scheduler/DAGScheduler.scala | 41 ++ .../apache/spark/scheduler/TaskSchedulerImpl.scala | 29 -- .../spark/deploy/DecommissionWorkerSuite.scala | 64 ++ .../spark/scheduler/TaskSchedulerImplSuite.scala | 36 +--- 6 files changed, 153 insertions(+), 37 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a36514e [3.0][SPARK-32518][CORE]
CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds
of resources
a36514e is described below
commit a36514e1fbf6920f0d18a7d8fb690822843eb2a2
Author: yi.wu
AuthorDate: Tue Aug 18 06:50:05 2020 +
[3.0][SPARK-32518][CORE]
CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds
of resources
### What changes were proposed in this pull request?
1. Make `CoarseGrainedSchedulerBackend.maxNumConcurrentTasks()` considers
all kinds of resources when calculating the max concurrent tasks
2. Refactor `calculateAvailableSlots()` to make it be able to be used for
both `CoarseGrainedSchedulerBackend` and `TaskSchedulerImpl`
### Why are the changes needed?
Currently, `CoarseGrainedSchedulerBackend.maxNumConcurrentTasks()` only
considers the CPU for the max concurrent tasks. This can cause the application
to hang when a barrier stage requires extra custom resources but the cluster
doesn't have enough corresponding resources. Because, without the checking for
other custom resources in `maxNumConcurrentTasks`, the barrier stage can be
submitted to the `TaskSchedulerImpl`. But the `TaskSchedulerImpl` won't launch
tasks for the barrier stage [...]
If the barrier stage doesn't launch all the tasks in one true, the
application will fail and suggest user to disable delay scheduling. However,
this actually a misleading suggestion since the real root cause is not enough
resources.
### Does this PR introduce _any_ user-facing change?
Yes. In case of a barrier stage requires more custom resources than the
cluster has, previously, the application will fail with misleading suggestion
of disabling delay scheduling. After this PR, the application will fail with
the error message saying not enough resources.
### How was this patch tested?
Added a unit test.
Closes #29395 from Ngone51/backport-spark-32518.
Authored-by: yi.wu
Signed-off-by: Wenchen Fan
---
.../main/scala/org/apache/spark/SparkContext.scala | 10 +++---
.../org/apache/spark/internal/config/Tests.scala | 15 +
.../scheduler/BarrierJobAllocationFailed.scala | 4 +--
.../org/apache/spark/scheduler/DAGScheduler.scala | 10 +++---
.../spark/scheduler/ExecutorResourceInfo.scala | 1 +
.../apache/spark/scheduler/SchedulerBackend.scala | 3 +-
.../apache/spark/scheduler/TaskSchedulerImpl.scala | 36 +-
.../cluster/CoarseGrainedSchedulerBackend.scala| 22 +++--
.../spark/BarrierStageOnSubmittedSuite.scala | 36 ++
.../scala/org/apache/spark/SparkContextSuite.scala | 2 ++
10 files changed, 124 insertions(+), 15 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 65c08cf..66fe1d7 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -1597,7 +1597,8 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get the max number of tasks that can be concurrent launched currently.
+ * Get the max number of tasks that can be concurrent launched based on the
resources
+ * could be used, even if some of them are being used at the moment.
* Note that please don't cache the value returned by this method, because
the number can change
* due to add/remove executors.
*
@@ -2776,8 +2777,9 @@ object SparkContext extends Logging {
}
// some cluster managers don't set the EXECUTOR_CORES config by default
(standalone
// and mesos coarse grained), so we can't rely on that config for those.
- val shouldCheckExecCores = executorCores.isDefined ||
sc.conf.contains(EXECUTOR_CORES) ||
+ var shouldCheckExecCores = executorCores.isDefined ||
sc.conf.contains(EXECUTOR_CORES) ||
(master.equalsIgnoreCase("yarn") || master.startsWith("k8s"))
+ shouldCheckExecCores &= !sc.conf.get(SKIP_VALIDATE_CORES_TESTING)
// Number of cores per executor must meet at least one task requirement.
if (shouldCheckExecCores && execCores < taskCores) {
@@ -2833,7 +2835,7 @@ object SparkContext extends Logging {
limitingResourceName = taskReq.resourceName
}
}
- if(!shouldCheckExecCores && Utils.isDynamicAllocationEnabled(sc.conf)) {
+ if(!shouldCheckExecCores) {
// if we can't rely on the executor cores config throw a warning for
user
logWarning("Please ensure that the number of slots available on your "
+
"executors is limited by the number
[spark] branch master updated (b33066f -> 1ac23de)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b33066f [SPARK-32622][SQL][TEST] Add case-sensitivity test for ORC predicate pushdown add 1ac23de [SPARK-32613][CORE] Fix regressions in DecommissionWorkerSuite No new revisions were added by this update. Summary of changes: .../executor/CoarseGrainedExecutorBackend.scala| 10 +++- .../org/apache/spark/internal/config/package.scala | 10 .../org/apache/spark/scheduler/DAGScheduler.scala | 41 ++ .../apache/spark/scheduler/TaskSchedulerImpl.scala | 29 -- .../spark/deploy/DecommissionWorkerSuite.scala | 64 ++ .../spark/scheduler/TaskSchedulerImplSuite.scala | 36 +--- 6 files changed, 153 insertions(+), 37 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [3.0][SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a36514e [3.0][SPARK-32518][CORE]
CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds
of resources
a36514e is described below
commit a36514e1fbf6920f0d18a7d8fb690822843eb2a2
Author: yi.wu
AuthorDate: Tue Aug 18 06:50:05 2020 +
[3.0][SPARK-32518][CORE]
CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds
of resources
### What changes were proposed in this pull request?
1. Make `CoarseGrainedSchedulerBackend.maxNumConcurrentTasks()` considers
all kinds of resources when calculating the max concurrent tasks
2. Refactor `calculateAvailableSlots()` to make it be able to be used for
both `CoarseGrainedSchedulerBackend` and `TaskSchedulerImpl`
### Why are the changes needed?
Currently, `CoarseGrainedSchedulerBackend.maxNumConcurrentTasks()` only
considers the CPU for the max concurrent tasks. This can cause the application
to hang when a barrier stage requires extra custom resources but the cluster
doesn't have enough corresponding resources. Because, without the checking for
other custom resources in `maxNumConcurrentTasks`, the barrier stage can be
submitted to the `TaskSchedulerImpl`. But the `TaskSchedulerImpl` won't launch
tasks for the barrier stage [...]
If the barrier stage doesn't launch all the tasks in one true, the
application will fail and suggest user to disable delay scheduling. However,
this actually a misleading suggestion since the real root cause is not enough
resources.
### Does this PR introduce _any_ user-facing change?
Yes. In case of a barrier stage requires more custom resources than the
cluster has, previously, the application will fail with misleading suggestion
of disabling delay scheduling. After this PR, the application will fail with
the error message saying not enough resources.
### How was this patch tested?
Added a unit test.
Closes #29395 from Ngone51/backport-spark-32518.
Authored-by: yi.wu
Signed-off-by: Wenchen Fan
---
.../main/scala/org/apache/spark/SparkContext.scala | 10 +++---
.../org/apache/spark/internal/config/Tests.scala | 15 +
.../scheduler/BarrierJobAllocationFailed.scala | 4 +--
.../org/apache/spark/scheduler/DAGScheduler.scala | 10 +++---
.../spark/scheduler/ExecutorResourceInfo.scala | 1 +
.../apache/spark/scheduler/SchedulerBackend.scala | 3 +-
.../apache/spark/scheduler/TaskSchedulerImpl.scala | 36 +-
.../cluster/CoarseGrainedSchedulerBackend.scala| 22 +++--
.../spark/BarrierStageOnSubmittedSuite.scala | 36 ++
.../scala/org/apache/spark/SparkContextSuite.scala | 2 ++
10 files changed, 124 insertions(+), 15 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 65c08cf..66fe1d7 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -1597,7 +1597,8 @@ class SparkContext(config: SparkConf) extends Logging {
}
/**
- * Get the max number of tasks that can be concurrent launched currently.
+ * Get the max number of tasks that can be concurrent launched based on the
resources
+ * could be used, even if some of them are being used at the moment.
* Note that please don't cache the value returned by this method, because
the number can change
* due to add/remove executors.
*
@@ -2776,8 +2777,9 @@ object SparkContext extends Logging {
}
// some cluster managers don't set the EXECUTOR_CORES config by default
(standalone
// and mesos coarse grained), so we can't rely on that config for those.
- val shouldCheckExecCores = executorCores.isDefined ||
sc.conf.contains(EXECUTOR_CORES) ||
+ var shouldCheckExecCores = executorCores.isDefined ||
sc.conf.contains(EXECUTOR_CORES) ||
(master.equalsIgnoreCase("yarn") || master.startsWith("k8s"))
+ shouldCheckExecCores &= !sc.conf.get(SKIP_VALIDATE_CORES_TESTING)
// Number of cores per executor must meet at least one task requirement.
if (shouldCheckExecCores && execCores < taskCores) {
@@ -2833,7 +2835,7 @@ object SparkContext extends Logging {
limitingResourceName = taskReq.resourceName
}
}
- if(!shouldCheckExecCores && Utils.isDynamicAllocationEnabled(sc.conf)) {
+ if(!shouldCheckExecCores) {
// if we can't rely on the executor cores config throw a warning for
user
logWarning("Please ensure that the number of slots available on your "
+
"executors is limited by the number
[spark] branch master updated (b33066f -> 1ac23de)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b33066f [SPARK-32622][SQL][TEST] Add case-sensitivity test for ORC predicate pushdown add 1ac23de [SPARK-32613][CORE] Fix regressions in DecommissionWorkerSuite No new revisions were added by this update. Summary of changes: .../executor/CoarseGrainedExecutorBackend.scala| 10 +++- .../org/apache/spark/internal/config/package.scala | 10 .../org/apache/spark/scheduler/DAGScheduler.scala | 41 ++ .../apache/spark/scheduler/TaskSchedulerImpl.scala | 29 -- .../spark/deploy/DecommissionWorkerSuite.scala | 64 ++ .../spark/scheduler/TaskSchedulerImplSuite.scala | 36 +--- 6 files changed, 153 insertions(+), 37 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b33066f -> 1ac23de)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b33066f [SPARK-32622][SQL][TEST] Add case-sensitivity test for ORC predicate pushdown add 1ac23de [SPARK-32613][CORE] Fix regressions in DecommissionWorkerSuite No new revisions were added by this update. Summary of changes: .../executor/CoarseGrainedExecutorBackend.scala| 10 +++- .../org/apache/spark/internal/config/package.scala | 10 .../org/apache/spark/scheduler/DAGScheduler.scala | 41 ++ .../apache/spark/scheduler/TaskSchedulerImpl.scala | 29 -- .../spark/deploy/DecommissionWorkerSuite.scala | 64 ++ .../spark/scheduler/TaskSchedulerImplSuite.scala | 36 +--- 6 files changed, 153 insertions(+), 37 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
