[
https://issues.apache.org/jira/browse/SPARK-50992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111097#comment-18111097
]
Ilya commented on SPARK-50992:
------------------------------
We can reproduce the OOM behavior described here on Apache Spark 4.1.3 in an
iterative DataFrame workload using AQE and cached DataFrames.
Environment:
* Apache Spark: 4.1.3
* Scala: 2.13.17
* Java: OpenJDK 17.0.19
* CatBoost Spark: ai.catboost:catboost-spark_4.1_2.13:1.2.10
With the default value of {{{}spark.sql.maxPlanStringLength{}}}, the Spark
4.1.3 driver eventually fails with:
{{java.lang.OutOfMemoryError: Java heap space}}
Changing only:
{{spark.sql.maxPlanStringLength=8192}}
prevents the driver OOM.
However, it does not fix the underlying planning-time degradation. The workload
continues running, but planning time grows very rapidly in later iterations.
For example, wall-clock time around construction of a downstream lazy
transform, with no Spark action executed inside the measured section, grows as:
step 11: 0.322 s
step 12: 0.703 s
step 13: 0.989 s
step 14: 1.570 s
step 15: 2.956 s
step 16: 5.797 s
step 17: 11.336 s
step 18: 22.606 s
In another run it continued to:
step 19: 45.476 s
step 20: 94.096 s
step 21: 179.974 s
step 22: 367.955 s
This A/B appears consistent with {{spark.sql.maxPlanStringLength}} mitigating
the memory impact of very large plan string representations while not
addressing the underlying large-plan / AQE planning overhead.
We also observe a Spark 3.4.3 -> 4.1.3 performance regression in the same
iterative cached-DataFrame workload. Disabling AQE only while materializing the
iterative cached state prevents the planning-time explosion while AQE remains
enabled for the rest of the application.
I am planning to report that behavior separately with a standalone synthetic
PySpark reproducer.
I can provide a sanitized OOM stack trace and additional per-iteration timing
data if they are useful.
> OOMs and performance issues with AQE in large plans
> ---------------------------------------------------
>
> Key: SPARK-50992
> URL: https://issues.apache.org/jira/browse/SPARK-50992
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.3, 3.5.4, 4.0.0
> Reporter: Ángel Álvarez Pascua
> Priority: Major
> Labels: pull-request-available
> Attachments: Main.scala
>
>
> When AQE is enabled, Spark triggers update events to the internal listener
> bus whenever a plan changes. These events include a plain-text description of
> the plan, which is computationally expensive to generate for large plans.
> {*}Key Issues:{*}{*}{{*}}
> *1. High Cost of Plan String Calculation:*
> * Generating the string description for large physical plans is a costly
> operation.
> * This impacts performance, particularly in complex workflows with frequent
> plan updates (e.g. persisting DataFrames).
> *2. Out-of-Memory (OOM) Errors:*
> * Events are stored in the listener bus as {{SQLExecutionUIData}} objects
> and retained until a threshold is reached.
> * This retention behavior can lead to memory exhaustion when processing
> large plans, causing OOM errors.
>
> *Current Workarounds Are Ineffective:*
> * *Reducing Retained Executions* ({{{}spark.sql.ui.retainedExecutions{}}}):
> Even when set to {{1}} or {{{}0{}}}, events are still created, requiring plan
> string calculations.
> * *Limiting Plan String Length* ({{{}spark.sql.maxPlanStringLength{}}}):
> Reducing the maximum string length (e.g., to {{{}1,000,000{}}}) may mitigate
> OOMs but does not eliminate the overhead of string generation.
> * *Available Explain Modes:* All existing explain modes are verbose and
> computationally expensive, failing to resolve these issues.
>
> *Proposed Solution:*
> Introduce a new explain mode, {*}{{off}}{*}, which suppresses the generation
> of plan string descriptions.
> * When this mode is enabled, Spark skips the calculation of plan
> descriptions altogether.
> * This resolves OOM errors and restores performance parity with non-AQE
> execution.
>
> *Impact of Proposed Solution:*
> * Eliminates OOMs in large plans with AQE enabled.
> * Reduces the performance overhead associated with plan string generation.
> * Ensures Spark scales better in environments with large, complex plans.
>
> *Reproducibility:*
> A test reproducing the issue has been attached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
