andygrove opened a new pull request, #6461: URL: https://github.com/apache/datafusion-comet/pull/6461
## Which issue does this PR close? Closes #6460. ## Rationale for this change Spark 4.2 added `BinaryType` to `Reverse`: `reverse(b)` on a binary column reverses its bytes and returns binary. `CometReverse` sent every argument that is not an array to the native string `reverse`, so on Spark 4.2 a query that reverses a binary column in a native plan failed with `Encountered non UTF-8 data`. ## What changes are included in this PR? `CometReverse` reports binary input as `Unsupported`. `CometReverse` is enrolled in the codegen dispatcher, so binary input now runs Spark's own `Reverse` inside the Comet plan, as arrays with binary, struct or map elements and collated strings already do, rather than falling back to Spark. It is `Unsupported` rather than `Incompatible` because there is no native path to opt into. With `allowIncompatible` on, an `Incompatible` binary input would still reach the native string `reverse`. Spark 4.1 and earlier cast binary to a string before `reverse`, so the new case never comes up there. The expressions page now lists binary input among the cases that go through the dispatcher. ## How are these changes tested? A new SQL file test, `expressions/string/reverse_binary.sql`, reverses a Parquet binary column on Spark 4.2 and later, with `allowIncompatible` off and on. It includes a value whose reversed bytes are not valid UTF-8. It fails on `main` with the native error and passes with this change. `reverse.sql`, `array_reverse.sql`, the 17 collation SQL tests and `GenerateDocsSuite` pass on Spark 4.2. `reverse.sql` and `array_reverse.sql` pass on 4.1, where the new test is skipped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
