cloud-fan commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1888717719
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
cloud-fan closed pull request #43954: [SPARK-46052][CORE] Remove function
TaskScheduler.killAllTaskAttempts
URL: https://github.com/apache/spark/pull/43954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
Ngone51 commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1888711672
@mridulm @cloud-fan Could you help merge the PR if you don't have other
comments? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1448451079
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -54,9 +54,9 @@ private[spark] trait TaskScheduler {
// Submit a sequence of tasks to
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1448203237
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -54,9 +54,9 @@ private[spark] trait TaskScheduler {
// Submit a sequence of tasks to
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1448202635
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1002,6 +1002,17 @@ private[spark] class TaskSetManager(
maybeFinishTaskSet()
}
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1448202496
##
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
##
@@ -296,18 +296,31 @@ private[spark] class TaskSchedulerImpl(
new
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1448192078
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -54,7 +54,7 @@ private[spark] trait TaskScheduler {
// Submit a sequence of tasks to
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1448181728
##
core/src/test/scala/org/apache/spark/TempLocalSparkContext.scala:
##
@@ -51,7 +51,7 @@ trait TempLocalSparkContext extends BeforeAndAfterEach
*/
def sc:
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1445733313
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -54,7 +54,7 @@ private[spark] trait TaskScheduler {
// Submit a sequence of tasks to
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1445731163
##
core/src/test/scala/org/apache/spark/TempLocalSparkContext.scala:
##
@@ -51,7 +51,7 @@ trait TempLocalSparkContext extends BeforeAndAfterEach
*/
def sc:
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1433910301
##
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##
@@ -54,12 +54,30 @@ import org.apache.spark.util.ArrayImplicits._
class
Ngone51 commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1864096635
@mridulm Add `DAGSchedulerAbortStageOffSuite` to test
`legacyAbortAfterCancelTasks = false`.
--
This is an automated message from the Apache Git Service.
To respond to the message,
Ngone51 commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1864073899
> Are we not testing when legacyAbortAfterCancelTasks == false ?
Like I mentioned at
https://github.com/apache/spark/pull/43954#discussion_r1420598351, since the
conf is static in
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1432410827
##
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##
@@ -54,12 +54,30 @@ import org.apache.spark.util.ArrayImplicits._
class
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1432407891
##
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##
@@ -54,12 +54,30 @@ import org.apache.spark.util.ArrayImplicits._
class
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1432407891
##
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##
@@ -54,12 +54,30 @@ import org.apache.spark.util.ArrayImplicits._
class
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r143248
##
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##
@@ -840,6 +856,7 @@ class DAGSchedulerSuite extends SparkFunSuite with
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1432382298
##
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##
@@ -840,6 +856,7 @@ class DAGSchedulerSuite extends SparkFunSuite with
Ngone51 commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1863983002
Hi @mridulm @cloud-fan I have moved "abort stage" outside of `cancelTasks`
again and fixed tests in another way (I previously fixed tests by moving "abort
stage" into `cancelTasks` 沈).
mridulm commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1853299046
Ah, interesting - I had not looked at barrier stage in as much detail; my
initial observation was it worked fine, but you are right - this does break the
assumption.
--
This is an
Ngone51 commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1851530021
Damn! Barrier stage seems to be a special case. It called
`killAllTaskAttempts()` to kill all the other tasks when there was a task
failure but didn't abort the stage as it would have a
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1422054329
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2603,4 +2603,14 @@ package object config {
.stringConf
.toSequence
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1421803481
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2603,4 +2603,14 @@ package object config {
.stringConf
.toSequence
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1420598351
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2603,4 +2603,13 @@ package object config {
.stringConf
.toSequence
Ngone51 commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1847130056
To fix the tests, I have to move
(https://github.com/apache/spark/pull/43954/commits/fe70ba95a67800b798443b8fa873d2b24efa2067)
the "abort stage" call back into `cancelTasks()` with the
Ngone51 commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1845378574
> Btw, we should flip this switch on and off in the relevant tests to check
if the behavior is preserved.
@mridulm I tried to enable
Ngone51 commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1843019152
Thanks @mridulm . Will address your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
mridulm commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1842129215
Thanks for the details @Ngone51, sorry for the delay in going over this -
your explaination makes sense to me.
Can you update the PR description about the fact that we are also fixing
Ngone51 commented on PR #43954:
URL: https://github.com/apache/spark/pull/43954#issuecomment-1838517185
@mridulm Thanks for the detailed comment.
> Additional call to `suspend` for existing `killAllTaskAttempts`
Note that we always call `markStageAsFinished` after the call to
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1411502671
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2603,4 +2603,13 @@ package object config {
.stringConf
.toSequence
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1410068855
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1871,21 +1871,6 @@ private[spark] class DAGScheduler(
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1410066008
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1894,24 +1894,8 @@ private[spark] class DAGScheduler(
job.numFinished
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409479495
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1894,24 +1894,8 @@ private[spark] class DAGScheduler(
job.numFinished
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409491574
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -2860,6 +2844,11 @@ private[spark] class DAGScheduler(
if
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409488053
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1871,21 +1871,6 @@ private[spark] class DAGScheduler(
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409488053
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1871,21 +1871,6 @@ private[spark] class DAGScheduler(
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409488053
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1871,21 +1871,6 @@ private[spark] class DAGScheduler(
mridulm commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409479495
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1894,24 +1894,8 @@ private[spark] class DAGScheduler(
job.numFinished
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409325407
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1004,6 +1004,17 @@ private[spark] class TaskSetManager(
maybeFinishTaskSet()
}
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409276985
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1004,6 +1004,17 @@ private[spark] class TaskSetManager(
maybeFinishTaskSet()
}
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409183926
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2603,4 +2603,13 @@ package object config {
.stringConf
.toSequence
beliefer commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1409052716
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1894,24 +1894,8 @@ private[spark] class DAGScheduler(
job.numFinished
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408994168
##
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
##
@@ -296,18 +296,31 @@ private[spark] class TaskSchedulerImpl(
new
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408984350
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1894,24 +1894,8 @@ private[spark] class DAGScheduler(
job.numFinished
beliefer commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408972767
##
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
##
@@ -296,18 +296,31 @@ private[spark] class TaskSchedulerImpl(
new
beliefer commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408964711
##
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:
##
@@ -1671,37 +1671,6 @@ class TaskSchedulerImplSuite extends SparkFunSuite with
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408809625
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1871,21 +1871,6 @@ private[spark] class DAGScheduler(
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408807471
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1871,21 +1871,6 @@ private[spark] class DAGScheduler(
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408807471
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1871,21 +1871,6 @@ private[spark] class DAGScheduler(
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408670937
##
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:
##
@@ -1671,37 +1671,6 @@ class TaskSchedulerImplSuite extends SparkFunSuite with
beliefer commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408664994
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -54,7 +54,7 @@ private[spark] trait TaskScheduler {
// Submit a sequence of tasks to
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408649384
##
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
##
@@ -296,18 +296,32 @@ private[spark] class TaskSchedulerImpl(
new
beliefer commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408625247
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -66,10 +66,6 @@ private[spark] trait TaskScheduler {
*/
def
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408624999
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -54,7 +54,7 @@ private[spark] trait TaskScheduler {
// Submit a sequence of tasks to
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408623095
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -2205,12 +2190,12 @@ private[spark] class DAGScheduler(
val message = s"Stage
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408622291
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2603,4 +2603,13 @@ package object config {
.stringConf
.toSequence
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408621909
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -66,10 +66,6 @@ private[spark] trait TaskScheduler {
*/
def killTaskAttempt(taskId:
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408583579
##
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
##
@@ -296,18 +296,32 @@ private[spark] class TaskSchedulerImpl(
new
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408582440
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -66,10 +66,6 @@ private[spark] trait TaskScheduler {
*/
def
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408582228
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -2205,12 +2190,12 @@ private[spark] class DAGScheduler(
val message = s"Stage
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1408581685
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2603,4 +2603,13 @@ package object config {
.stringConf
.toSequence
beliefer commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1407378513
##
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##
@@ -54,7 +54,7 @@ private[spark] trait TaskScheduler {
// Submit a sequence of tasks to
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1407349896
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -2179,12 +2164,12 @@ private[spark] class DAGScheduler(
val message = s"Stage
cloud-fan commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1402889739
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -2179,12 +2164,12 @@ private[spark] class DAGScheduler(
val message = s"Stage
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1402869071
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -2179,12 +2164,12 @@ private[spark] class DAGScheduler(
val message = s"Stage
JoshRosen commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1402598488
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1871,21 +1871,6 @@ private[spark] class DAGScheduler(
JoshRosen commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1402597413
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -2179,12 +2164,12 @@ private[spark] class DAGScheduler(
val message = s"Stage
JoshRosen commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1402526009
##
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
##
@@ -302,10 +302,19 @@ private[spark] class TaskSchedulerImpl(
reason: String):
Ngone51 commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1401687380
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -1871,21 +1871,6 @@ private[spark] class DAGScheduler(
Ngone51 opened a new pull request, #43954:
URL: https://github.com/apache/spark/pull/43954
### What changes were proposed in this pull request?
This PR removes the interface `TaskScheduler.killAllTaskAttempts` and its
implementations. And replace it with
71 matches
Mail list logo
