Boyang Jerry Peng created SPARK-61517:
-----------------------------------------

             Summary: Fall back to regular shuffles for non-root top-K with a 
hidden shuffle
                 Key: SPARK-61517
                 URL: https://issues.apache.org/jira/browse/SPARK-61517
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Boyang Jerry Peng


A local batch query containing a non-root {{TakeOrderedAndProjectExec}} 
followed by a shuffle can fail when SQL local pipelined shuffles are enabled. 
The top-K operator creates a regular single-partition RDD shuffle internally 
when its input has multiple partitions. The SQL enabling rules do not see that 
hidden shuffle and mark the outer exchange as pipelined. DAGScheduler 
subsequently rejects the mixed graph because the regular prefix is not already 
materialized.

This was identified while reviewing [SPARK-60271 / PR 
#59523|https://github.com/apache/spark/pull/59523#discussion_r4518327978]. The 
failure mechanism predates that PR and involves no caching or checkpointing.

h3. Reproduction

Run the following Scala code using a fresh SparkContext. Repeat in another 
fresh context with {{aqe = true}}; both settings reproduce the failure. The 
incremental manager must be configured before SparkContext creation.

{code:scala}
import org.apache.spark.sql.SparkSession

val aqe = false // Repeat with true in a fresh SparkContext.
val spark = SparkSession.builder()
  .master("local[16]")
  .appName("top-k-hidden-shuffle")
  .config("spark.shuffle.manager.incremental",
    "org.apache.spark.shuffle.local.pipelined.PipelinedChannelShuffleManager")
  .config("spark.shuffle.channel.batchSize", "1024")
  .config("spark.shuffle.channel.queueCapacity", "64")
  .config("spark.sql.shuffle.localPipelined.enabled", "true")
  .config("spark.sql.adaptive.enabled", aqe.toString)
  .config("spark.speculation", "false")
  .config("spark.sql.shuffle.partitions", "4")
  .getOrCreate()

val df = spark.range(0, 4000, 1, 2)
  .orderBy("id")
  .limit(10)
  .repartition(2)

df.explain()
val result = df.collect().sorted.toSeq
assert(result == (0L until 10L))
{code}

h3. Expected behavior

The query returns IDs 0 through 9. When pipelined execution cannot support the 
actual RDD graph, the SQL enabling rules should leave the query on regular 
shuffles.

h3. Actual behavior

With either AQE setting, the initial physical plan has the following shape 
(inside an {{AdaptiveSparkPlan}} when AQE is enabled):

{code:text}
Exchange RoundRobinPartitioning(2), REPARTITION_BY_NUM, isPipelined=true
+- TakeOrderedAndProject(limit=10, orderBy=[id ASC NULLS FIRST], output=[id])
   +- Range (0, 4000, step=1, splits=2)
{code}

{{collect()}} fails with:

{code:text}
A job that mixes a pipelined shuffle dependency with a regular shuffle 
dependency is only supported when every regular shuffle is a fully-materialized 
prefix below the pipelined shuffles; otherwise a job must be either all-regular 
or all-pipelined.
{code}

h3. Cause

{{TakeOrderedAndProjectExec.doExecute}} calls {{child.execute()}} and, when the 
resulting RDD has more than one partition, creates a {{ShuffledRowRDD}} using 
{{prepareShuffleDependency(..., SinglePartition, ...)}}. That dependency 
defaults to a regular shuffle. It is not represented by a separate 
{{ShuffleExchangeExec}} below the top-K node.

The resulting RDD graph is:

{code:text}
Range (2 partitions)
  -> per-partition top-K candidates
  -> regular shuffle (1 partition, not yet materialized)
  -> global top-K
  -> pipelined shuffle (2 partitions)
  -> collect
{code}

The visible-exchange ancestry checks in {{EnablePipelinedShuffle}} and 
{{AQEEnablePipelinedShuffle}} miss the hidden regular shuffle. This is not a 
lack of available local task slots: the reproduction uses {{local[16]}}.

Relevant source at the tested PR head:

* 
[TakeOrderedAndProjectExec.doExecute|https://github.com/apache/spark/blob/e110f064df23202a0bb2f033251da7eb31da6f58/sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala#L347-L373]
* 
[EnablePipelinedShuffle|https://github.com/apache/spark/blob/e110f064df23202a0bb2f033251da7eb31da6f58/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnablePipelinedShuffle.scala]
* 
[AQEEnablePipelinedShuffle|https://github.com/apache/spark/blob/e110f064df23202a0bb2f033251da7eb31da6f58/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEEnablePipelinedShuffle.scala]
* [Shared 
PipelinedShuffleEligibility|https://github.com/apache/spark/blob/e110f064df23202a0bb2f033251da7eb31da6f58/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/PipelinedShuffleEligibility.scala]

h3. Verified scope and workaround

* The exact reproduction above fails with AQE disabled and enabled at PR #59523 
head {{e110f064df23202a0bb2f033251da7eb31da6f58}}.
* A separately constructed Dataset with the SQL local-pipelining flag disabled 
succeeds for both AQE settings and returns IDs 0 through 9.
* An earlier non-AQE probe using {{spark.range(0, 1000, 1, 
2).orderBy(col("id").desc).limit(10).repartition(2).collect()}} failed with the 
same mixed-shuffle rejection at both the PR merge base 
{{ec018f385fd4aab72933658c976a1bd2c8f7fcb7}} and the PR head. This establishes 
the pre-existing mechanism; it does not establish the first affected release.

Workaround: set {{spark.sql.shuffle.localPipelined.enabled=false}} *before 
constructing a fresh Dataset* for the query. Changing the flag does not rewrite 
an already constructed physical plan.

h3. Proposed direction and validation

Evaluate a shared eligibility boundary for {{TakeOrderedAndProjectExec}}, used 
by both SQL enabling rules, so the plan retains regular exchanges when top-K 
may create a hidden shuffle. An unconditional boundary is simple but also 
removes surrounding pipelining for one-partition top-K inputs, where no hidden 
shuffle is needed; assess that tradeoff against a safely determined 
width-sensitive check.

Add end-to-end regression tests with AQE on and off that execute the 
reproduction, verify IDs 0 through 9, and verify regular exchanges. Include a 
one-partition top-K case to document the chosen eligibility policy. Preserve 
supported shuffle-free producer shapes outside this top-K restriction.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to