andygrove opened a new issue, #206:
URL: https://github.com/apache/datafusion-spark/issues/206
Part of #177 and apache/datafusion-comet#7300.
### What the function does
`upper(str)` returns `str` with every character converted to uppercase.
`ucase` is a registry alias
(`expression[Upper]("ucase", true)`). Available since Spark 1.0.1.
The input type is `STRING` (any collation from 4.0 on, including trim
collations). `Upper` mixes in
`ImplicitCastInputTypes`, so the analyzer casts other atomic inputs
(numbers, binary, dates, and so on) to `STRING`
before the function runs. The kernel only has to handle strings. The return
type is the input's string type
(collation kept), the function is null-intolerant (NULL in, NULL out), and
it is deterministic.
The conversion is a full Unicode case mapping on code points, so the output
can be longer than the input
(`ß` becomes `SS`). ASCII-only input takes a fast path that maps each byte
separately.
### Examples
```sql
SELECT upper('SparkSql');
-- SPARKSQL
SELECT ucase('SparkSql');
-- SPARKSQL
SELECT upper('äbćδe');
-- ÄBĆΔE
SELECT upper('ß fi ffi ff st ῗ');
-- SS FI FFI FF ST Ϊ͂ (the last one is U+0399 U+0308 U+0342)
SELECT upper('θαλασσινος'); -- final sigma
-- ΘΑΛΑΣΣΙΝΟΣ
SELECT upper('ςαλατα');
-- ΣΑΛΑΤΑ
SELECT upper('i\u0307onic'); -- i + U+0307 combining dot above
-- İONIC (I + U+0307, not U+0130)
SELECT upper('İonic'); -- U+0130 is already uppercase
-- İONIC
SELECT upper('𐐭𝔸'); -- surrogate pairs: Deseret small letter, math
double-struck A
-- 𐐅𝔸
SELECT upper('a🙃B🙃c');
-- A🙃B🙃C
SELECT upper(NULL);
-- NULL
```
(All non-trivial outputs come from `CollationSupportSuite.testUpper`. The
suite checks both the ICU and the JVM
code paths and expects the same result from each.)
### Edge cases to test
- NULL input returns NULL. Empty string returns empty string.
- Scalar and array input. Also test `Utf8`, `LargeUtf8` and `Utf8View`.
- One-to-many mappings that make the output longer: `ß` -> `SS`, ligatures
`fi`/`ffi`/`st`, `ῗ` -> 3 code points.
The output can have more bytes than the input, so don't size the output
buffer from the input.
- Turkish dotted/dotless I: `İ` (U+0130) stays `İ`, `ı` (U+0131) -> `I`, `i`
-> `I`. Spark uses root-locale rules,
not Turkish rules (see the locale note under Implementation notes).
- Greek sigma: `σ` and `ς` both map to `Σ`.
- Supplementary-plane characters (4-byte UTF-8 / UTF-16 surrogate pairs):
`𐐭` -> `𐐅`. `𝔸` and emoji are unchanged.
- Input that is all ASCII and input that mixes ASCII with other characters.
Spark takes a per-byte fast path only when
the whole string is ASCII.
- Binary and numeric inputs reach the function only after the analyzer casts
them to string, e.g. `upper(123)` is
`upper(CAST(123 AS STRING))`. A binary-to-string cast in Spark does not
validate UTF-8, but Spark's slow path decodes
through `java.lang.String` / `toValidString()`, so invalid bytes come out
as U+FFFD. Arrow `Utf8` input is always
valid UTF-8.
- Collations (4.0+): `UTF8_BINARY` (default), `UTF8_LCASE`, `UNICODE`,
`UNICODE_CI` and locale collations. For
ICU locale collations, non-ASCII input uses that collation's locale (for
example Turkish casing under a `tr`
collation). All-ASCII input always takes the locale-independent ASCII fast
path, even under a `tr` collation. A first
version can support only `UTF8_BINARY` and return an error for other
collations.
- `spark.sql.ansi.enabled` has no effect.
### Differences across Spark versions
- 3.4 / 3.5: `UTF8String.toUpperCase()`. All-ASCII input uses a per-byte
fast path. Everything else falls back to
`java.lang.String.toUpperCase()`, which uses the JVM's default locale.
There are no collations.
- 4.0+: dispatches through `CollationSupport.Upper.exec(v, collationId,
useICU)`. A new config,
`spark.sql.icu.caseMappings.enabled` (default `true`, since 4.0.0), picks
ICU `UCharacter.toUpperCase` instead
of the JVM mapping for `UTF8_BINARY`. `UTF8_LCASE` always uses ICU. Other
collations use ICU with the collation's
locale. With the config set to `false`, the behavior matches 3.x.
- 4.1 / 4.2: no behavior change found.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
(`Upper`, which uses `String2StringExpression`),
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java`
(`Upper`),
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java`
(`toUpperCase`), and
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`
(`toUpperCase`, `toUpperCaseAscii`).
- DataFusion core has `upper` (`datafusion/functions/src/string/upper.rs`,
`string/common.rs::to_upper`). It uses
Rust's `char::to_uppercase`, which is the locale-insensitive, full
(one-to-many) Unicode mapping on code points. For
`UTF8_BINARY` that is the same model as Spark's default ICU root mapping,
so `ß` -> `SS`, `fi` -> `FI` and
surrogate pairs all behave the same. The remaining differences are:
- Unicode version: Rust std, ICU4J and the JDK each ship their own tables.
- Spark's dependence on the JVM default locale. With a `tr`, `az` or `lt`
default locale, Spark gives Turkish or
Lithuanian results for non-ASCII strings.
- Collation-specific locales.
The UDF can probably wrap DataFusion's kernel. Most of the work is tests
that pin these cases, plus a decision on
how to handle non-default collations.
- Module: `crates/spark/src/function/string/` (`upper.rs`), registered as
`upper`. Expose `ucase` through
`aliases()`.
- Tests: add `crates/sqllogictest/test_files/string/upper.slt` (see
`elt.slt` and the other files in that directory)
covering the examples above. Document the function with `#[user_doc]` and
run `dev/update_function_docs.sh`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]