rdtr commented on code in PR #13518:
URL: https://github.com/apache/gluten/pull/13518#discussion_r4572586005
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -165,6 +170,16 @@ object Validators {
}
}
+ private object FallbackByUnsupportedSparkFeatures extends Validator {
+ override def validate(plan: SparkPlan): Validator.OutCome = plan match {
+ // The scan reports `SinglePartition` and Spark may have elided a
shuffle above it, while the
+ // native scan doesn't coalesce its splits into a single task.
+ case p: FileSourceScanExec if
SparkShimLoader.getSparkShims.isSingleTaskExecutionScan(p) =>
Review Comment:
Doesn't `LocalTableScanExec` have the same single-task gap?
In Spark 4.3 it also reports `SinglePartition` when marked
(`LocalTableScanExec.scala:79-85`) and creates one empty partition for empty
input (`:62-65`). `VeloxLocalTableScanTransformer` keeps
`plan.outputPartitioning` (`VeloxLocalTableScanTransformer.scala:152`) but
parallelizes into several slices (`:124`) and returns `emptyRDD` for empty
input (`:114`).
With `spark.gluten.sql.columnar.localTableScan=true` and single-task
execution enabled, a global aggregation with the shuffle elided could return
several rows, or none on empty input. Both configs are off by default, which is
why "empty local relation + global aggregation returns one row" passes in
`GlutenMarkSingleTaskExecutionSuite` in #13520.
A follow-up could handle it in the transformer rather than with a fallback:
when `outputPartitioning` is `SinglePartition`, parallelize into one slice and
use `parallelize(Seq.empty, 1)` for empty input, as `LocalTableScanExec` does
(`:59-72`). That keeps the scan native, and checking `outputPartitioning` needs
no shim, since it is always `UnknownPartitioning(0)` before 4.3.
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
}
}
+ private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+ override def computeValue(clazz: Class[_]): Method =
clazz.getMethod("shuffleBlockResolver")
Review Comment:
Nit: `clazz.getMethod("shuffleBlockResolver")` returns the implementing
class's own method, and `invoke` throws `IllegalAccessException` if that class
isn't public (a Java anonymous class, for example). Looking the method up once
on the declaring type (`BlockingShuffleManager` on 4.3, `ShuffleManager`
before) avoids that and drops the `ClassValue`. Fine to leave as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]