JacobZheng created SPARK-38682:
----------------------------------
Summary: Complex calculations with lead to driver oom
Key: SPARK-38682
URL: https://issues.apache.org/jira/browse/SPARK-38682
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.0
Reporter: JacobZheng
My spark job is working fine in version 3.0.1. After I upgraded to 3.2, the
driver would hang during runtime due to oom. The dump file shows that the
stageMetrics in SQLAppStatusListener are taking up a lot of memory.I'm
wondering if it's related to the SPARK-33016 change, or if the execution plan
change has created more stages, causing the driver to run out of memory, or
some other reason.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
