[
https://issues.apache.org/jira/browse/SPARK-38682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719333#comment-17719333
]
Nicolas Fraison commented on SPARK-38682:
-----------------------------------------
We are observing same pattern on one of our test job trying to migrate from
spark 3.1.1 to 3.3.1.
Looking at the SQL query UI we discover that for lost of queries the number of
executed jobs has drastically increase.
With 3.1.1:
!311.png|width=935,height=289!
With 3.3.1:
!331.png|width=931,height=415!
We have tried running the job with spark.sql.adaptive.enabled false but this
doesn't change much the behaviour.
Most of those "new" jobs are some broadcast exchange which could be explain as
the tested dataset is a little one.
Disabling the join broadcast (spark.sql.autoBroadcastJoinThreshold set to -1)
reduce number of jobs but leads to quite slow test.
Is there any idea on which change that could have lead to such number of jobs?
> Complex calculations with lead to driver oom
> ---------------------------------------------
>
> Key: SPARK-38682
> URL: https://issues.apache.org/jira/browse/SPARK-38682
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: JacobZheng
> Priority: Major
> Attachments: 311.png, 331.png, screenshot-1.png
>
>
> My spark job is working fine in version 3.0.1. After I upgraded to 3.2, the
> driver would hang during runtime due to oom. The dump file shows that the
> stageMetrics in SQLAppStatusListener are taking up a lot of memory.I'm
> wondering if it's related to the SPARK-33016 change, or if the execution plan
> change has created more tasks, causing the driver to run out of memory, or
> some other reason.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
