andygrove opened a new issue, #206:
URL: https://github.com/apache/datafusion-spark/issues/206

   Part of #177 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `upper(str)` returns `str` with every character converted to uppercase. 
`ucase` is a registry alias
   (`expression[Upper]("ucase", true)`). Available since Spark 1.0.1.
   
   The input type is `STRING` (any collation from 4.0 on, including trim 
collations). `Upper` mixes in
   `ImplicitCastInputTypes`, so the analyzer casts other atomic inputs 
(numbers, binary, dates, and so on) to `STRING`
   before the function runs. The kernel only has to handle strings. The return 
type is the input's string type
   (collation kept), the function is null-intolerant (NULL in, NULL out), and 
it is deterministic.
   
   The conversion is a full Unicode case mapping on code points, so the output 
can be longer than the input
   (`ß` becomes `SS`). ASCII-only input takes a fast path that maps each byte 
separately.
   
   ### Examples
   
   ```sql
   SELECT upper('SparkSql');
   -- SPARKSQL
   SELECT ucase('SparkSql');
   -- SPARKSQL
   SELECT upper('äbćδe');
   -- ÄBĆΔE
   SELECT upper('ß fi ffi ff st ῗ');
   -- SS FI FFI FF ST Ϊ͂      (the last one is U+0399 U+0308 U+0342)
   SELECT upper('θαλασσινος');   -- final sigma
   -- ΘΑΛΑΣΣΙΝΟΣ
   SELECT upper('ςαλατα');
   -- ΣΑΛΑΤΑ
   SELECT upper('i\u0307onic');  -- i + U+0307 combining dot above
   -- İONIC                 (I + U+0307, not U+0130)
   SELECT upper('İonic');        -- U+0130 is already uppercase
   -- İONIC
   SELECT upper('𐐭𝔸');          -- surrogate pairs: Deseret small letter, math 
double-struck A
   -- 𐐅𝔸
   SELECT upper('a🙃B🙃c');
   -- A🙃B🙃C
   SELECT upper(NULL);
   -- NULL
   ```
   
   (All non-trivial outputs come from `CollationSupportSuite.testUpper`. The 
suite checks both the ICU and the JVM
   code paths and expects the same result from each.)
   
   ### Edge cases to test
   
   - NULL input returns NULL. Empty string returns empty string.
   - Scalar and array input. Also test `Utf8`, `LargeUtf8` and `Utf8View`.
   - One-to-many mappings that make the output longer: `ß` -> `SS`, ligatures 
`fi`/`ffi`/`st`, `ῗ` -> 3 code points.
     The output can have more bytes than the input, so don't size the output 
buffer from the input.
   - Turkish dotted/dotless I: `İ` (U+0130) stays `İ`, `ı` (U+0131) -> `I`, `i` 
-> `I`. Spark uses root-locale rules,
     not Turkish rules (see the locale note under Implementation notes).
   - Greek sigma: `σ` and `ς` both map to `Σ`.
   - Supplementary-plane characters (4-byte UTF-8 / UTF-16 surrogate pairs): 
`𐐭` -> `𐐅`. `𝔸` and emoji are unchanged.
   - Input that is all ASCII and input that mixes ASCII with other characters. 
Spark takes a per-byte fast path only when
     the whole string is ASCII.
   - Binary and numeric inputs reach the function only after the analyzer casts 
them to string, e.g. `upper(123)` is
     `upper(CAST(123 AS STRING))`. A binary-to-string cast in Spark does not 
validate UTF-8, but Spark's slow path decodes
     through `java.lang.String` / `toValidString()`, so invalid bytes come out 
as U+FFFD. Arrow `Utf8` input is always
     valid UTF-8.
   - Collations (4.0+): `UTF8_BINARY` (default), `UTF8_LCASE`, `UNICODE`, 
`UNICODE_CI` and locale collations. For
     ICU locale collations, non-ASCII input uses that collation's locale (for 
example Turkish casing under a `tr`
     collation). All-ASCII input always takes the locale-independent ASCII fast 
path, even under a `tr` collation. A first
     version can support only `UTF8_BINARY` and return an error for other 
collations.
   - `spark.sql.ansi.enabled` has no effect.
   
   ### Differences across Spark versions
   
   - 3.4 / 3.5: `UTF8String.toUpperCase()`. All-ASCII input uses a per-byte 
fast path. Everything else falls back to
     `java.lang.String.toUpperCase()`, which uses the JVM's default locale. 
There are no collations.
   - 4.0+: dispatches through `CollationSupport.Upper.exec(v, collationId, 
useICU)`. A new config,
     `spark.sql.icu.caseMappings.enabled` (default `true`, since 4.0.0), picks 
ICU `UCharacter.toUpperCase` instead
     of the JVM mapping for `UTF8_BINARY`. `UTF8_LCASE` always uses ICU. Other 
collations use ICU with the collation's
     locale. With the config set to `false`, the behavior matches 3.x.
   - 4.1 / 4.2: no behavior change found.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
     (`Upper`, which uses `String2StringExpression`),
     
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java`
 (`Upper`),
     
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java`
 (`toUpperCase`), and
     
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java` 
(`toUpperCase`, `toUpperCaseAscii`).
   - DataFusion core has `upper` (`datafusion/functions/src/string/upper.rs`, 
`string/common.rs::to_upper`). It uses
     Rust's `char::to_uppercase`, which is the locale-insensitive, full 
(one-to-many) Unicode mapping on code points. For
     `UTF8_BINARY` that is the same model as Spark's default ICU root mapping, 
so `ß` -> `SS`, `fi` -> `FI` and
     surrogate pairs all behave the same. The remaining differences are:
     - Unicode version: Rust std, ICU4J and the JDK each ship their own tables.
     - Spark's dependence on the JVM default locale. With a `tr`, `az` or `lt` 
default locale, Spark gives Turkish or
       Lithuanian results for non-ASCII strings.
     - Collation-specific locales.
   
     The UDF can probably wrap DataFusion's kernel. Most of the work is tests 
that pin these cases, plus a decision on
     how to handle non-default collations.
   - Module: `crates/spark/src/function/string/` (`upper.rs`), registered as 
`upper`. Expose `ucase` through
     `aliases()`.
   - Tests: add `crates/sqllogictest/test_files/string/upper.slt` (see 
`elt.slt` and the other files in that directory)
     covering the examples above. Document the function with `#[user_doc]` and 
run `dev/update_function_docs.sh`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to