utkarsh39 opened a new pull request, #44321:
URL: https://github.com/apache/spark/pull/44321
### What changes were proposed in this pull request?
`AccumulableInfo` is one of the top heap consumers in driver's heap dumps
for stages with many tasks. For a stage with a large number o
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428552502
##
core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala:
##
@@ -75,14 +75,47 @@ class TaskInfo(
* accumulable to be updated multiple times in a single ta
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428553296
##
core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala:
##
@@ -75,14 +75,47 @@ class TaskInfo(
* accumulable to be updated multiple times in a single ta
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428555609
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -846,11 +850,49 @@ private[spark] class TaskSetManager(
// "result.value()" in "Tas
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428556859
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -787,6 +787,8 @@ private[spark] class TaskSetManager(
// SPARK-37300: when the task
JoshRosen commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428629120
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -846,11 +850,49 @@ private[spark] class TaskSetManager(
// "result.value()" in "Tas
JoshRosen commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428629362
##
core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala:
##
@@ -75,14 +75,47 @@ class TaskInfo(
* accumulable to be updated multiple times in a single ta
JoshRosen commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428630112
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -787,6 +787,8 @@ private[spark] class TaskSetManager(
// SPARK-37300: when the task
JoshRosen commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428630951
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -846,11 +850,49 @@ private[spark] class TaskSetManager(
// "result.value()" in "Tas
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428631597
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -846,11 +850,49 @@ private[spark] class TaskSetManager(
// "result.value()" in "Tas
mridulm commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1858692524
I have not looked into this in a lot of detail (and given my vacation plans,
might not be able to unfortunately).
Will do one note for @cloud-fan and @JoshRosen - perhaps you have analy
beliefer commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428673442
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -846,11 +850,49 @@ private[spark] class TaskSetManager(
// "result.value()" in "Task
beliefer commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428673442
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -846,11 +850,49 @@ private[spark] class TaskSetManager(
// "result.value()" in "Task
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428689443
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -846,11 +850,49 @@ private[spark] class TaskSetManager(
// "result.value()" in "Tas
cloud-fan commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1858720574
@mridulm `LiveTask` gets the `TaskInfo` via the listener, this PR sends the
original `TaskInfo` instance to `DAGScheduler` and thus to the event bus. We
keep the cloned `TaskInfo` with
beliefer commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1428771003
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -846,11 +850,49 @@ private[spark] class TaskSetManager(
// "result.value()" in "Task
mridulm commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1859050296
@cloud-fan thanks for checking !
Note that this does not apply to `executorLost` case though (I will call it
out in the exact location) - since the `Resubmitted` event will now have
in
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1429033491
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1429033491
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1429775174
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1429775174
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1430850168
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager(
addPendingTask(index)
utkarsh39 commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1864719908
**Proposal To Gain Consensus**
The PR alleviates memory pressure on the driver although at the cost of
introducing a breaking change as identified by @JoshRosen in
https://github.co
mridulm commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1866052989
Sounds good to me, thoughts @JoshRosen, @cloud-fan ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
cloud-fan commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1866163422
SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscrib
JoshRosen commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1866794659
The proposed "make the behavior change optional and off-by-default with
option for users to opt-in" approach sounds reasonable to me: users or
platforms that don't rely on the hopefully
utkarsh39 commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1874648077
Disabled the changes by default @JoshRosen @mridulm. Can you all PTAL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1441419344
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -846,11 +851,49 @@ private[spark] class TaskSetManager(
// "result.value()" in "TaskR
utkarsh39 commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1443107674
##
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala:
##
@@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with
LocalSparkCon
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1443157668
##
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala:
##
@@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with
LocalSparkConte
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1443157668
##
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala:
##
@@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with
LocalSparkConte
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1443157668
##
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala:
##
@@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with
LocalSparkConte
utkarsh39 commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1445001773
##
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala:
##
@@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with
LocalSparkCon
utkarsh39 commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1881845026
@mridulm Can you PTAL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comme
mridulm commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1448069167
##
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala:
##
@@ -643,6 +657,29 @@ class SparkListenerSuite extends SparkFunSuite with
LocalSparkConte
utkarsh39 commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1448110832
##
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala:
##
@@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with
LocalSparkCon
cloud-fan commented on code in PR #44321:
URL: https://github.com/apache/spark/pull/44321#discussion_r1448875469
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2620,4 +2620,14 @@ package object config {
.stringConf
.toSequence
cloud-fan closed pull request #44321: [SPARK-46383] Reduce Driver Heap Usage by
Reducing the Lifespan of `TaskInfo.accumulables()`
URL: https://github.com/apache/spark/pull/44321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub an
cloud-fan commented on PR #44321:
URL: https://github.com/apache/spark/pull/44321#issuecomment-1888334658
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
46 matches
Mail list logo