felipepessoto commented on code in PR #13510:
URL: https://github.com/apache/gluten/pull/13510#discussion_r4606302517


##########
backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/stats/GlutenDeltaJobStatsTracker.scala:
##########
@@ -115,12 +137,46 @@ object GlutenDeltaJobStatsTracker extends Logging {
       new GlutenDeltaJobStatsFallbackTracker(tracker)
   }
 
+  private def createStatsPlan(
+      dataCols: Seq[Attribute],
+      aggregates: Seq[AggregateExpression]): SparkPlan = {
+    val statsAttrs = 
aggregates.flatMap(_.aggregateFunction.aggBufferAttributes)
+    val statsResultAttrs = 
aggregates.flatMap(_.aggregateFunction.inputAggBufferAttributes)
+    val aggOp = SortAggregateExec(
+      None,
+      isStreaming = false,
+      None,
+      Seq.empty,
+      aggregates,
+      statsAttrs,
+      0,
+      statsResultAttrs,
+      StatisticsInputNode(dataCols)
+    )
+    val projOp = ProjectExec(statsResultAttrs, aggOp)
+    val offloads = Seq(OffloadOthers()).map(_.toStrcitRule())
+    val validatorBuilder: GlutenConfig => Validator = conf =>
+      Validators.newValidator(conf, offloads)
+    val config = GlutenConfig.get
+    val transformRule =
+      HeuristicTransform.WithRewrites(validatorBuilder(config), 
Seq(PullOutPreProject), offloads)
+    ColumnarCollapseTransformStages(config)(transformRule(projOp))

Review Comment:
   You’re right to question whether we still need this. The original motivation 
was the NTZ statistics-support gap, which #12967 addressed. I haven’t 
identified another unsupported statistics pattern under default settings, so 
I’m not sure we should proceed with this PR in its current form.
   
   The remaining reproductions are configuration-driven. For example, disabling 
native aggregation while keeping native writing enabled makes the statistics 
planner leave a Spark aggregate, but the tracker still assumes a fully native 
plan. That causes a column-count mismatch; disabling projection instead causes 
the unchecked `ProjectExec` cast to fail. Explicitly enabling NTZ validation 
also reproduces rejection in the direct statistics test.
   
   This PR checks the complete stage—not just its root—and uses Spark 
statistics when it cannot fully offload, while retaining native file writing. 
It therefore handles those rejected plans safely rather than adding missing 
native support.
   
   Before proceeding, could we clarify whether ordinary-query offload switches 
should also govern the writer’s internal statistics? If yes, this fixes crashes 
for that configuration. If not, statistics-specific planning rules would 
address those cases more directly, and we could defer the general fallback 
change. I’ve updated #12538 to distinguish the historical motivation from this 
remaining question.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to