Copilot commented on code in PR #13519:
URL: https://github.com/apache/gluten/pull/13519#discussion_r4439503201
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -165,6 +170,16 @@ object Validators {
}
}
+ private object FallbackByUnsupportedSparkFeatures extends Validator {
+ override def validate(plan: SparkPlan): Validator.OutCome = plan match {
+ // The scan reports `SinglePartition` and Spark may have elided a
shuffle above it, while the
+ // native scan doesn't coalesce its splits into a single task.
+ case p: FileSourceScanExec if
SparkShimLoader.getSparkShims.isSingleTaskExecutionScan(p) =>
+ fail(s"${p.nodeName} uses single-task execution, which is not
supported")
+ case _ => pass()
Review Comment:
With single-task optimization and local-table-scan offload enabled on Spark
4.3, `LocalTableScanExec` can report `SinglePartition` but pass this validator.
The native replacement preserves that declaration while creating multiple
partitions (or zero for empty input), so a global aggregation whose shuffle was
omitted can return per-partition results or no result. Also fall back for
single-partition local scans, or make their native execution honor the
single-partition contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
