LuciferYang opened a new pull request, #13343: URL: https://github.com/apache/gluten/pull/13343
## What changes are proposed in this pull request? `VeloxIntermediateData` maps regr_r2's Velox intermediate row to Spark's buffer attributes with Corr's order, because `RegrR2` extends `PearsonCorrelation`. Velox's regr_r2 has a different layout, (c2, count, m2X, meanX, meanY, m2Y) with X being the second argument, and Spark's `RegrR2(y, x)` passes its arguments to `PearsonCorrelation(y, x)`, so Spark's `xAvg` and `xMk` describe y. Plans that run one regr_r2 phase in Velox and another in vanilla Spark returned wrong results. When every phase runs in Velox, the two directions of the mapping cancel out. This adds a RegrR2 order, `ck, n, yMk, yAvg, xAvg, xMk`, matched before the PearsonCorrelation case. The intermediate types are the same for both functions, so the type mapping is unchanged. The Bolt backend has the same mapping in `BoltIntermediateData`. It is left for a follow-up because no CI job runs the Bolt tests. ## How was this patch tested? Added two tests to `VeloxAggregateFunctionsSuite`. Both fail without this change, and both check that the plan has a Velox aggregation and a vanilla Spark aggregation computing regr_r2. - Velox partial, vanilla final: `histogram_numeric(distinct z, 3)` is not offloaded, so the last two of the four aggregations Spark plans for a single DISTINCT run in vanilla Spark and merge the buffer Velox produced. - Vanilla partial, Velox final: the partial aggregation is forced to fall back. This test only compares groups where x and y both vary. For a constant x or y, Velox's final regr_r2 result matches Spark only from SPARK-55969 on, so it differs from every Spark version Gluten builds against except 4.2, whichever mapping is used. ## Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Opus. Related issue: #13340. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
