Eren Avsarogullari created SPARK-47148:
------------------------------------------
Summary: [AQE] Avoid to materialize ShuffleQueryStage on the
cancellation
Key: SPARK-47148
URL: https://issues.apache.org/jira/browse/SPARK-47148
Project: Spark
Issue Type: Bug
Components: Shuffle, SQL
Affects Versions: 4.0.0
Reporter: Eren Avsarogullari
AQE can materialize *ShuffleQueryStage* on the cancellation. This causes
unnecessary stage materialization by submitting Shuffle Job. Under normal
circumstances, if the stage is already non-materialized (a.k.a
ShuffleQueryStage.shuffleFuture is not initialized yet), it should just be
skipped without materializing it.
Please find sample use-case:
*1- Stage Materialization Steps:*
When stage materialization is failed:
{code:java}
1.1- ShuffleQueryStage1 - is materialized successfully,
1.2- ShuffleQueryStage2 - materialization is failed,
1.3- ShuffleQueryStage3 - Not materialized yet so
ShuffleQueryStage3.shuffleFuture is not initialized yet{code}
*2- Stage Cancellation Steps:*
{code:java}
2.1- ShuffleQueryStage1 - is canceled due to already materialized,
2.2- ShuffleQueryStage2 - is earlyFailedStage so currently, it is skipped as
default because it could not be materialized,
2.3- ShuffleQueryStage3 - Problem is here: This stage is not materialized yet
but currently, it is also tried to cancel and this stage requires to be
materialized first.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
