[PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

utkarsh39 opened a new pull request, #44321: URL: https://github.com/apache/spark/pull/44321 ### What changes were proposed in this pull request? `AccumulableInfo` is one of the top heap consumers in driver's heap dumps for stages with many tasks. For a stage with a large number o

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428552502 ## core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala: ## @@ -75,14 +75,47 @@ class TaskInfo( * accumulable to be updated multiple times in a single ta

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428553296 ## core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala: ## @@ -75,14 +75,47 @@ class TaskInfo( * accumulable to be updated multiple times in a single ta

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428555609 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -846,11 +850,49 @@ private[spark] class TaskSetManager( // "result.value()" in "Tas

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428556859 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -787,6 +787,8 @@ private[spark] class TaskSetManager( // SPARK-37300: when the task

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

JoshRosen commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428629120 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -846,11 +850,49 @@ private[spark] class TaskSetManager( // "result.value()" in "Tas

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

JoshRosen commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428629362 ## core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala: ## @@ -75,14 +75,47 @@ class TaskInfo( * accumulable to be updated multiple times in a single ta

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

JoshRosen commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428630112 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -787,6 +787,8 @@ private[spark] class TaskSetManager( // SPARK-37300: when the task

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

JoshRosen commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428630951 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -846,11 +850,49 @@ private[spark] class TaskSetManager( // "result.value()" in "Tas

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428631597 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -846,11 +850,49 @@ private[spark] class TaskSetManager( // "result.value()" in "Tas

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1858692524 I have not looked into this in a lot of detail (and given my vacation plans, might not be able to unfortunately). Will do one note for @cloud-fan and @JoshRosen - perhaps you have analy

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

beliefer commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428673442 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -846,11 +850,49 @@ private[spark] class TaskSetManager( // "result.value()" in "Task

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

beliefer commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428673442 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -846,11 +850,49 @@ private[spark] class TaskSetManager( // "result.value()" in "Task

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428689443 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -846,11 +850,49 @@ private[spark] class TaskSetManager( // "result.value()" in "Tas

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1858720574 @mridulm `LiveTask` gets the `TaskInfo` via the listener, this PR sends the original `TaskInfo` instance to `DAGScheduler` and thus to the event bus. We keep the cloned `TaskInfo` with

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

beliefer commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1428771003 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -846,11 +850,49 @@ private[spark] class TaskSetManager( // "result.value()" in "Task

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1859050296 @cloud-fan thanks for checking ! Note that this does not apply to `executorLost` case though (I will call it out in the exact location) - since the `Resubmitted` event will now have in

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1429033491 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1429033491 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1429775174 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1429775174 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1430656022 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1430850168 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1086,7 +1133,7 @@ private[spark] class TaskSetManager( addPendingTask(index)

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

utkarsh39 commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1864719908 **Proposal To Gain Consensus** The PR alleviates memory pressure on the driver although at the cost of introducing a breaking change as identified by @JoshRosen in https://github.co

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1866052989 Sounds good to me, thoughts @JoshRosen, @cloud-fan ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1866163422 SGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

JoshRosen commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1866794659 The proposed "make the behavior change optional and off-by-default with option for users to opt-in" approach sounds reasonable to me: users or platforms that don't rely on the hopefully

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

utkarsh39 commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1874648077 Disabled the changes by default @JoshRosen @mridulm. Can you all PTAL? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1441419344 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -846,11 +851,49 @@ private[spark] class TaskSetManager( // "result.value()" in "TaskR

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

utkarsh39 commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1443107674 ## core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala: ## @@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with LocalSparkCon

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1443157668 ## core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala: ## @@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with LocalSparkConte

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1443157668 ## core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala: ## @@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with LocalSparkConte

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1443157668 ## core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala: ## @@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with LocalSparkConte

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

utkarsh39 commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1445001773 ## core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala: ## @@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with LocalSparkCon

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

utkarsh39 commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1881845026 @mridulm Can you PTAL? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

mridulm commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1448069167 ## core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala: ## @@ -643,6 +657,29 @@ class SparkListenerSuite extends SparkFunSuite with LocalSparkConte

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

utkarsh39 commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1448110832 ## core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala: ## @@ -289,6 +290,17 @@ class SparkListenerSuite extends SparkFunSuite with LocalSparkCon

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on code in PR #44321: URL: https://github.com/apache/spark/pull/44321#discussion_r1448875469 ## core/src/main/scala/org/apache/spark/internal/config/package.scala: ## @@ -2620,4 +2620,14 @@ package object config { .stringConf .toSequence

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan closed pull request #44321: [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` URL: https://github.com/apache/spark/pull/44321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()` [spark]

cloud-fan commented on PR #44321: URL: https://github.com/apache/spark/pull/44321#issuecomment-1888334658 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c