rdtr opened a new pull request, #13585:
URL: https://github.com/apache/gluten/pull/13585

   Depends on #13520. Only the last commit belongs to this PR: 
https://github.com/rdtr/incubator-gluten/commit/92431899136475ae2a91058d45a1cfca3e629c2f
   
   ## What changes were proposed in this pull request?
   
   Spark 4.3 (SPARK-58213) returns NULL from `corr` on a zero-variance column 
in ANSI mode, which is what Velox already does. Spark 4.2 and earlier throw 
`DIVIDE_BY_ZERO` there (#13255).
   
   - `MiscOperatorSuite` "Support multi-children count with row construct": 
ANSI mode is now enabled on Spark 4.3+ and still disabled before that. 
`ansiFallback` is turned off so the query is offloaded.
   - `VeloxAggregateFunctionsSuite`: new test "corr with zero variance column 
in ANSI mode", ANSI on, runs only on Spark 4.3+ 
(`assume(isSparkVersionGE("4.3"))`), checks the result against vanilla Spark 
and that `HashAggregateExecTransformer` is used. It is added here because 
`MiscOperatorSuite` is ignored on Spark 4.2+ (GLUTEN-13179), so the first 
change alone is not exercised on 4.3 yet.
   
   Spark 4.3.0 contains SPARK-58213 (`3a07a3dbb0e`), 4.2.1-rc3 does not.
   
   Relates to #13255. This covers the Spark 4.3 half of that issue; the 
4.2-and-earlier behavior difference is still open, so I would keep the issue 
open.
   
   ## How was this patch tested?
   
   `mvn test -Pspark-4.3 -Pbackends-velox -pl backends-velox` with 
`VeloxAggregateFunctionsDefaultSuite`, "corr with zero variance column in ANSI 
mode": 1 passed. Without `ansiFallback=false` the same test fails because every 
operator falls back.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] [GLUTEN-13517][VL] Run t... rdtr (via GitHub) via commits

Reply via email to