andygrove opened a new issue, #219:
URL: https://github.com/apache/datafusion-spark/issues/219
Part of #178 and apache/datafusion-comet#7300.
### What the function does
`reverse(expr)` reverses its input, and the result has the same type as the
input:
- A STRING is reversed character by character, by Unicode code point.
- An ARRAY has its elements reversed. `containsNull` is unchanged, and NULL
elements are kept in their mirrored
positions.
- A BINARY value (Spark 4.2.0+) has its bytes reversed.
The input types are
`TypeCollection(StringTypeWithCollation(supportsTrimCollation = true),
BinaryType, ArrayType)`
with `ImplicitCastInputTypes`. Any other atomic type is implicitly cast to
STRING first, so
`reverse(3215)` returns the string `'5123'`. Struct and map inputs fail
analysis with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` (requiredType `("STRING" or
"BINARY" or "ARRAY")` in 4.2). The
expression is null-intolerant, so a NULL input gives NULL.
The string form exists since Spark 1.5.0, the array form since 2.4.0, and
the binary form since 4.2.0. There are
no aliases. The function is a single overloaded name in Spark, so the
datafusion-spark UDF must accept all three
input kinds.
### Examples
```sql
SELECT reverse('Spark SQL');
-- LQS krapS
SELECT reverse(array(2, 1, 4, 3));
-- [3,4,1,2]
SELECT hex(reverse(x'CAFE'));
-- FECA (Spark 4.2.0+)
SELECT reverse(3215);
-- 5123
SELECT reverse(null);
-- NULL
SELECT reverse(array(array(1, 2), array(3, 4)));
-- [[3,4],[1,2]]
SELECT reverse(struct(1, 'a'));
-- error DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE
```
Spark's test suites also cover these array cases
(`CollectionExpressionsSuite`):
- `[1, null, 2, null]` becomes `[null, 2, null, 1]`.
- `[null, null, null]` stays the same.
- `[]` stays `[]`.
### Edge cases to test
- NULL string, binary and array inputs, plus an empty string, an empty
binary and an empty array.
- Unicode:
- Spark's `UTF8String.reverse` copies whole UTF-8 sequences (code points),
so multi-byte characters and
4-byte characters (emoji, supplementary-plane CJK) stay intact.
- Grapheme clusters are not preserved: a base letter followed by a
combining mark comes out with the mark
before the letter, and ZWJ emoji sequences are split.
- Test a decomposed `e` + U+0301 and a flag emoji.
- Binary input, from `DataFrameFunctionsSuite` /
`CollectionExpressionsSuite`:
- `x'CAFE'` becomes `x'FECA'`, `x''` stays `x''`, and `x'01'` stays
`x'01'`.
- `x'E687'` becomes `x'87E6'`. That input is a valid UTF-8 prefix, but the
bytes are reversed raw.
- The result type must stay BINARY.
- Arrays:
- NULL elements, nested arrays (only the outer level is reversed), and
struct, map and binary elements.
- `containsNull = false` arrays.
- Non-string scalar inputs (INT, DOUBLE, DATE, TIMESTAMP, DECIMAL) are cast
to STRING with Spark's cast
formatting before reversing. The cast output depends on the session time
zone for TIMESTAMP.
- Collations (4.0+): the result keeps the input's collation, and a trim
collation is accepted. The reversal
itself ignores the collation.
- Scalar and column inputs, including sliced arrays.
### Differences across Spark versions
- 3.4 and 3.5: the input types are `TypeCollection(StringType, ArrayType)`.
BINARY is not a member, so a BINARY
argument is implicitly cast to STRING (both TypeCoercion and
AnsiTypeCoercion allow atomic-to-string), and the
result is a STRING. The binary bytes are interpreted as UTF-8 and reversed
by sequence.
- 4.0 and 4.1: the same, plus collation-aware string input types.
- 4.2: `BinaryType` is added to the input types, so `reverse(binary)`
returns BINARY with the bytes reversed.
Engines that target several Spark versions need a way to choose between
these behaviours, or the planner
must insert the cast for older versions.
### Implementation notes
- Spark implements this in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala`
(`Reverse`). The string path is `UTF8String.reverse()` in
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`.
- DataFusion core splits this into two functions:
- `reverse` (`datafusion/functions/src/unicode/reverse.rs`) reverses by
`char` (code point), which matches
Spark for valid UTF-8. It only handles string types and implicitly
coerces any input to string, so arrays
and binary are not supported.
- `array_reverse` / `list_reverse`
(`datafusion/functions-nested/src/reverse.rs`) handles `List`,
`LargeList`, `FixedSizeList` and `ListView`, keeps the type, and matches
Spark's element semantics.
- The datafusion-spark UDF can dispatch on the input type:
- strings go to DataFusion's string `reverse`,
- lists go to `array_reverse`,
- binary needs a new byte-reverse kernel for `Binary`, `LargeBinary` and
`BinaryView`.
- In `coerce_types`, cast other scalar types to `Utf8` the way Spark does,
using Spark cast formatting
(for example `2.0` for DOUBLE).
- There is already a placeholder
`crates/sqllogictest/test_files/collection/reverse.slt`, ported from Sail with
its queries commented out. Turn it into real tests, put the UDF in
`crates/spark/src/function/collection/`
(next to `size`), and document it with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]