[ https://issues.apache.org/jira/browse/SPARK-47148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eren Avsarogullari updated SPARK-47148: --------------------------------------- Summary: Avoid to materialize AQE ExchangeQueryStageExec on the cancellation (was: Avoid to materialize AQE QueryStages on the cancellation) > Avoid to materialize AQE ExchangeQueryStageExec on the cancellation > ------------------------------------------------------------------- > > Key: SPARK-47148 > URL: https://issues.apache.org/jira/browse/SPARK-47148 > Project: Spark > Issue Type: Bug > Components: Shuffle, SQL > Affects Versions: 4.0.0 > Reporter: Eren Avsarogullari > Priority: Major > Labels: pull-request-available > > AQE can materialize *ShuffleQueryStage* on the cancellation. This causes > unnecessary stage materialization by submitting Shuffle Job. Under normal > circumstances, if the stage is already non-materialized (a.k.a > ShuffleQueryStage.shuffleFuture is not initialized yet), it should just be > skipped without materializing it. > Please find sample use-case: > *1- Stage Materialization Steps:* > When stage materialization is failed: > {code:java} > 1.1- ShuffleQueryStage1 - is materialized successfully, > 1.2- ShuffleQueryStage2 - materialization is failed, > 1.3- ShuffleQueryStage3 - Not materialized yet so > ShuffleQueryStage3.shuffleFuture is not initialized yet{code} > *2- Stage Cancellation Steps:* > {code:java} > 2.1- ShuffleQueryStage1 - is canceled due to already materialized, > 2.2- ShuffleQueryStage2 - is earlyFailedStage so currently, it is skipped as > default by AQE because it could not be materialized, > 2.3- ShuffleQueryStage3 - Problem is here: This stage is not materialized yet > but currently, it is also tried to cancel and this stage requires to be > materialized first.{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org