andygrove opened a new issue, #208:
URL: https://github.com/apache/datafusion-spark/issues/208
Part of #177 and apache/datafusion-comet#7300.
### What the function does
`initcap(str)` returns `str` with the first letter of each word in titlecase
and every other letter in lowercase.
For the default `UTF8_BINARY` collation, a word starts at the beginning of
the string or right after an ASCII space
(U+0020). Tabs, newlines, punctuation and other whitespace do not start a
word. Available since Spark 1.5.0.
The input type is `STRING` (any collation from 4.0 on). `InitCap` mixes in
`ImplicitCastInputTypes`, so the analyzer
casts other atomic inputs to `STRING`. The return type is the input's string
type (collation kept), the function is
null-intolerant, and it is deterministic.
With the 4.0+ default (`spark.sql.icu.caseMappings.enabled=true`), each
word's first code point goes through ICU
titlecase, which is not the same as uppercase (`dž` -> `Dž`) and can produce
several code points (`ß` -> `Ss`). The rest
of the string is lowercased one code point at a time. Capital sigma becomes
final or non-final depending on the
surrounding characters, and `İ` becomes `i` + U+0307.
### Examples
```sql
SELECT initcap('sPark sql');
-- Spark Sql
SELECT initcap('hi THOMAS');
-- Hi Thomas
SELECT initcap('a.b,c'); -- only ASCII space separates words
-- A.b,c
SELECT initcap('a. b-c');
-- A. B-c
SELECT initcap('a🙃B🙃c'); -- emoji is not a word separator
-- A🙃b🙃c
SELECT initcap('aB 世 de');
-- Ab 世 De
SELECT initcap('ΘΑΛΑΣΣΙΝΟΣ'); -- word-final sigma
-- Θαλασσινος
SELECT initcap('džaba Ljubav NJegova'); -- titlecase digraphs, not uppercase
-- Džaba Ljubav Njegova
SELECT initcap('ß fi ffi ff st ῗ'); -- ICU default; the last word is U+0399
U+0308 U+0342
-- Ss Fi Ffi Ff St Ϊ͂
SELECT initcap('FIDELİO');
-- Fideli̇o (i + U+0307)
SELECT initcap('İo'); -- ICU default
-- İo
SELECT initcap('𐐭𝔸'); -- surrogate pair, ICU default
-- 𐐅𝔸
SELECT initcap(NULL);
-- NULL
```
(The outputs come from `ExpressionDescription`, `postgreSQL/strings.sql.out`
and
`CollationSupportSuite.testInitCap` with `UTF8_BINARY` and ICU mappings on.)
With `spark.sql.icu.caseMappings.enabled=false` (and in Spark 3.x), the
suite expects different results:
`initcap('ß fi ffi ff st ῗ')` -> `ß fi ffi ff st ῗ`, `initcap('İo')` -> `İo` (I +
U+0307), and `initcap('𐐭𝔸')` -> `𐐭𝔸`.
### Edge cases to test
- NULL returns NULL. Empty string returns empty string. Test scalar and
array input, and `Utf8`, `LargeUtf8` and
`Utf8View`.
- Word boundaries: several spaces in a row, leading and trailing spaces, and
tab or newline as a separator.
`initcap('a\tb')` gives `A\tb`, because only U+0020 counts (derived from
`toTitleCaseICU` and
`toTitleCaseAscii`). Hyphens, apostrophes and digits are not separators
either: `initcap('o''neil')` -> `O'neil`.
- All-ASCII fast path: Spark lowercases the whole string and then uppercases
each byte that follows a space.
- Titlecase that differs from uppercase: `dz`/`DZ`/`Dz` give `Dz` at the start
of a word and `dz` elsewhere.
`initcap('dzDZDz')` -> `Dzdzdz`.
- One-to-many titlecase at the start of a word: `ß` -> `Ss`, `fi` -> `Fi`,
`ffi` -> `Ffi`, `ῗ` -> 3 code points.
- Greek sigma inside a word becomes `ς` or `σ` from context.
`initcap('θαλασσινοσ')` -> `Θαλασσινοσ`, because a
lowercase `σ` is kept.
- Capital I with dot above inside a word: `İ` -> `i` + U+0307. At the start
of a word it stays `İ`.
- Supplementary-plane characters: with ICU on, `𐐭` at the start of a word
becomes `𐐅`. The JVM path titlecases
UTF-16 units one at a time and leaves surrogate pairs unchanged.
- Non-ASCII input that Spark decodes from invalid UTF-8 (after `CAST(binary
AS STRING)`): Spark calls `makeValid`
first, so invalid bytes become U+FFFD.
- Collations (4.0+): `UTF8_LCASE`, `UNICODE` and ICU collations use ICU
`UCharacter.toTitleCase` with a word
`BreakIterator`, so the word boundaries change. `initcap('a.b,c' COLLATE
UNICODE)` -> `A.b,C`,
`initcap('a?b世c' COLLATE UNICODE)` -> `A?B世C`, and `initcap('a🙃B🙃c'
COLLATE UTF8_LCASE)` -> `A🙃B🙃C`. A first
version can support only `UTF8_BINARY`.
- `spark.sql.ansi.enabled` has no effect.
### Differences across Spark versions
- 3.4 / 3.5: `UTF8String.toLowerCase().toTitleCase()`. The lowercase step is
`java.lang.String.toLowerCase()` with the
JVM default locale. The titlecase step applies
`Character.toTitleCase(char)` to each UTF-16 unit that follows `' '`.
It never expands to several characters (`ß` stays `ß`) and it leaves
surrogate pairs unchanged.
- 4.0: adds `spark.sql.icu.caseMappings.enabled` (default `true`, since
4.0.0). With the default, `UTF8_BINARY` uses
`CollationAwareUTF8String.toTitleCaseICU`, which changes results for
one-to-many characters, `İ`, and supplementary
characters (see the examples). Setting the config to `false` restores the
3.x behavior. Collation support is also
new in 4.0.
- 4.1 / 4.2: no behavior change found. No other legacy config affects
`initcap`.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
(`InitCap`),
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java`
(`InitCap`), and
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java`
(`toTitleCaseICU`,
`appendTitleCasedCodepoint`, `appendLowerCasedGreekCapitalSigma`,
`codepointOneToManyTitleCaseLookupTable`). The
ASCII fast path is `UTF8String.toTitleCaseAscii`.
- DataFusion core has `initcap`
(`datafusion/functions/src/unicode/initcap.rs`), but it is not compatible:
- Words are split at any non-alphanumeric character (`is_alphanumeric`),
so DataFusion returns `A.B,C` for
`'a.b,c'` and `A🙃B🙃C` for `'a🙃B🙃c'`.
- The first letter goes through `char::to_uppercase`, not titlecase, so
`dž` -> `DŽ` and `ß` -> `SS`.
- The rest of each word is lowercased one character at a time with
`char::to_lowercase`, so there is no final-sigma
handling.
Write a new kernel. Rust's `char::to_lowercase` / `to_uppercase` cover
most of the mapping. Titlecase needs the
Unicode titlecase mapping (simple `Titlecase_Mapping` plus the
`SpecialCasing` titlecase entries, about 48 code
points that expand). One option is a small lookup table for characters
whose titlecase differs from their uppercase.
Another is an existing crate such as `unicode-case-mapping`, if adding the
dependency is acceptable. Final sigma needs
the `Cased` / `Case_Ignorable` properties, used the same way as in
`followedByCasedLetter`.
- Consider a mode flag for the 3.x / `icu.caseMappings.enabled=false`
behavior: lowercase the whole string, then
titlecase only BMP characters after a space with no expansion.
- Module: `crates/spark/src/function/string/` (`initcap.rs`).
- Tests: add `crates/sqllogictest/test_files/string/initcap.slt` (follow
`elt.slt`) with the cases above. Document the
function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]