[ https://issues.apache.org/jira/browse/SPARK-38682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Fraison updated SPARK-38682: ------------------------------------ Attachment: 331.png > Complex calculations with lead to driver oom > --------------------------------------------- > > Key: SPARK-38682 > URL: https://issues.apache.org/jira/browse/SPARK-38682 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.2.0 > Reporter: JacobZheng > Priority: Major > Attachments: 311.png, 331.png, screenshot-1.png > > > My spark job is working fine in version 3.0.1. After I upgraded to 3.2, the > driver would hang during runtime due to oom. The dump file shows that the > stageMetrics in SQLAppStatusListener are taking up a lot of memory.I'm > wondering if it's related to the SPARK-33016 change, or if the execution plan > change has created more tasks, causing the driver to run out of memory, or > some other reason. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org