dongjoon-hyun commented on code in PR #56928:
URL: https://github.com/apache/spark/pull/56928#discussion_r3510620488
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -744,10 +774,28 @@ case class FileSourceScanExec(
inputRDD :: Nil
}
+ /**
+ * The input RDD, coalesced to a single partition when this scan runs in
single-task mode. This
+ * enforces the `SinglePartition` output partitioning reported by
`outputPartitioning`, which is
+ * estimated from the statically-selected files and may not correspond
exactly to the number of
+ * partitions the input RDD produces after dynamic pruning. Coalescing here
keeps the query
+ * correct in either case.
+ */
+ private[spark] lazy val maybeCoalesceInputRDD: RDD[InternalRow] = {
+ if (useSingleTaskExecution && inputRDD.getNumPartitions > 1) {
+ inputRDD.coalesce(1)
Review Comment:
Is this consistent with `outputPartitioning` code path? In
`outputPartitioning` code, `bucketedScan` is handled before
`useSingleTaskExecution`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
