andygrove opened a new issue, #208:
URL: https://github.com/apache/datafusion-spark/issues/208

   Part of #177 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `initcap(str)` returns `str` with the first letter of each word in titlecase 
and every other letter in lowercase.
   For the default `UTF8_BINARY` collation, a word starts at the beginning of 
the string or right after an ASCII space
   (U+0020). Tabs, newlines, punctuation and other whitespace do not start a 
word. Available since Spark 1.5.0.
   
   The input type is `STRING` (any collation from 4.0 on). `InitCap` mixes in 
`ImplicitCastInputTypes`, so the analyzer
   casts other atomic inputs to `STRING`. The return type is the input's string 
type (collation kept), the function is
   null-intolerant, and it is deterministic.
   
   With the 4.0+ default (`spark.sql.icu.caseMappings.enabled=true`), each 
word's first code point goes through ICU
   titlecase, which is not the same as uppercase (`dž` -> `Dž`) and can produce 
several code points (`ß` -> `Ss`). The rest
   of the string is lowercased one code point at a time. Capital sigma becomes 
final or non-final depending on the
   surrounding characters, and `İ` becomes `i` + U+0307.
   
   ### Examples
   
   ```sql
   SELECT initcap('sPark sql');
   -- Spark Sql
   SELECT initcap('hi THOMAS');
   -- Hi Thomas
   SELECT initcap('a.b,c');            -- only ASCII space separates words
   -- A.b,c
   SELECT initcap('a. b-c');
   -- A. B-c
   SELECT initcap('a🙃B🙃c');          -- emoji is not a word separator
   -- A🙃b🙃c
   SELECT initcap('aB 世 de');
   -- Ab 世 De
   SELECT initcap('ΘΑΛΑΣΣΙΝΟΣ');       -- word-final sigma
   -- Θαλασσινος
   SELECT initcap('džaba Ljubav NJegova'); -- titlecase digraphs, not uppercase
   -- Džaba Ljubav Njegova
   SELECT initcap('ß fi ffi ff st ῗ');      -- ICU default; the last word is U+0399 
U+0308 U+0342
   -- Ss Fi Ffi Ff St Ϊ͂
   SELECT initcap('FIDELİO');
   -- Fideli̇o            (i + U+0307)
   SELECT initcap('İo');               -- ICU default
   -- İo
   SELECT initcap('𐐭𝔸');              -- surrogate pair, ICU default
   -- 𐐅𝔸
   SELECT initcap(NULL);
   -- NULL
   ```
   
   (The outputs come from `ExpressionDescription`, `postgreSQL/strings.sql.out` 
and
   `CollationSupportSuite.testInitCap` with `UTF8_BINARY` and ICU mappings on.)
   
   With `spark.sql.icu.caseMappings.enabled=false` (and in Spark 3.x), the 
suite expects different results:
   `initcap('ß fi ffi ff st ῗ')` -> `ß fi ffi ff st ῗ`, `initcap('İo')` -> `İo` (I + 
U+0307), and `initcap('𐐭𝔸')` -> `𐐭𝔸`.
   
   ### Edge cases to test
   
   - NULL returns NULL. Empty string returns empty string. Test scalar and 
array input, and `Utf8`, `LargeUtf8` and
     `Utf8View`.
   - Word boundaries: several spaces in a row, leading and trailing spaces, and 
tab or newline as a separator.
     `initcap('a\tb')` gives `A\tb`, because only U+0020 counts (derived from 
`toTitleCaseICU` and
     `toTitleCaseAscii`). Hyphens, apostrophes and digits are not separators 
either: `initcap('o''neil')` -> `O'neil`.
   - All-ASCII fast path: Spark lowercases the whole string and then uppercases 
each byte that follows a space.
   - Titlecase that differs from uppercase: `dz`/`DZ`/`Dz` give `Dz` at the start 
of a word and `dz` elsewhere.
     `initcap('dzDZDz')` -> `Dzdzdz`.
   - One-to-many titlecase at the start of a word: `ß` -> `Ss`, `fi` -> `Fi`, 
`ffi` -> `Ffi`, `ῗ` -> 3 code points.
   - Greek sigma inside a word becomes `ς` or `σ` from context. 
`initcap('θαλασσινοσ')` -> `Θαλασσινοσ`, because a
     lowercase `σ` is kept.
   - Capital I with dot above inside a word: `İ` -> `i` + U+0307. At the start 
of a word it stays `İ`.
   - Supplementary-plane characters: with ICU on, `𐐭` at the start of a word 
becomes `𐐅`. The JVM path titlecases
     UTF-16 units one at a time and leaves surrogate pairs unchanged.
   - Non-ASCII input that Spark decodes from invalid UTF-8 (after `CAST(binary 
AS STRING)`): Spark calls `makeValid`
     first, so invalid bytes become U+FFFD.
   - Collations (4.0+): `UTF8_LCASE`, `UNICODE` and ICU collations use ICU 
`UCharacter.toTitleCase` with a word
     `BreakIterator`, so the word boundaries change. `initcap('a.b,c' COLLATE 
UNICODE)` -> `A.b,C`,
     `initcap('a?b世c' COLLATE UNICODE)` -> `A?B世C`, and `initcap('a🙃B🙃c' 
COLLATE UTF8_LCASE)` -> `A🙃B🙃C`. A first
     version can support only `UTF8_BINARY`.
   - `spark.sql.ansi.enabled` has no effect.
   
   ### Differences across Spark versions
   
   - 3.4 / 3.5: `UTF8String.toLowerCase().toTitleCase()`. The lowercase step is 
`java.lang.String.toLowerCase()` with the
     JVM default locale. The titlecase step applies 
`Character.toTitleCase(char)` to each UTF-16 unit that follows `' '`.
     It never expands to several characters (`ß` stays `ß`) and it leaves 
surrogate pairs unchanged.
   - 4.0: adds `spark.sql.icu.caseMappings.enabled` (default `true`, since 
4.0.0). With the default, `UTF8_BINARY` uses
     `CollationAwareUTF8String.toTitleCaseICU`, which changes results for 
one-to-many characters, `İ`, and supplementary
     characters (see the examples). Setting the config to `false` restores the 
3.x behavior. Collation support is also
     new in 4.0.
   - 4.1 / 4.2: no behavior change found. No other legacy config affects 
`initcap`.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
 (`InitCap`),
     
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java`
 (`InitCap`), and
     
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java`
 (`toTitleCaseICU`,
     `appendTitleCasedCodepoint`, `appendLowerCasedGreekCapitalSigma`, 
`codepointOneToManyTitleCaseLookupTable`). The
     ASCII fast path is `UTF8String.toTitleCaseAscii`.
   - DataFusion core has `initcap` 
(`datafusion/functions/src/unicode/initcap.rs`), but it is not compatible:
     - Words are split at any non-alphanumeric character (`is_alphanumeric`), 
so DataFusion returns `A.B,C` for
       `'a.b,c'` and `A🙃B🙃C` for `'a🙃B🙃c'`.
     - The first letter goes through `char::to_uppercase`, not titlecase, so 
`dž` -> `DŽ` and `ß` -> `SS`.
     - The rest of each word is lowercased one character at a time with 
`char::to_lowercase`, so there is no final-sigma
       handling.
   
     Write a new kernel. Rust's `char::to_lowercase` / `to_uppercase` cover 
most of the mapping. Titlecase needs the
     Unicode titlecase mapping (simple `Titlecase_Mapping` plus the 
`SpecialCasing` titlecase entries, about 48 code
     points that expand). One option is a small lookup table for characters 
whose titlecase differs from their uppercase.
     Another is an existing crate such as `unicode-case-mapping`, if adding the 
dependency is acceptable. Final sigma needs
     the `Cased` / `Case_Ignorable` properties, used the same way as in 
`followedByCasedLetter`.
   - Consider a mode flag for the 3.x / `icu.caseMappings.enabled=false` 
behavior: lowercase the whole string, then
     titlecase only BMP characters after a space with no expansion.
   - Module: `crates/spark/src/function/string/` (`initcap.rs`).
   - Tests: add `crates/sqllogictest/test_files/string/initcap.slt` (follow 
`elt.slt`) with the cases above. Document the
     function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to