andygrove opened a new issue, #199:
URL: https://github.com/apache/datafusion-spark/issues/199

   Part of #177 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `translate(input, from, to)` replaces every character of `input` that 
appears in `from` with the character at the same position in `to`. Available 
since Spark 1.5.0.
   
   - Characters are Unicode code points. Supplementary characters (surrogate 
pairs in Java) are handled as single characters.
   - If `from` is longer than `to`, the extra `from` characters are deleted 
from `input`.
   - If `to` is longer, its extra characters are ignored.
   - If a character appears more than once in `from`, its first occurrence 
decides the mapping.
   - Spark builds the dictionary with U+0000 as the "delete" sentinel. As a 
result, any `from` character explicitly mapped to U+0000 in `to` is also 
deleted, not replaced with NUL.
   
   All three arguments are STRING 
(`StringTypeNonCSAICollation(supportsTrimCollation = true)`) via 
`ImplicitCastInputTypes`, so numbers and similar values are cast to STRING. The 
return type is `input`'s type. It is null-intolerant: NULL in any argument 
returns NULL.
   
   ### Examples
   
   ```sql
   SELECT translate('AaBbCc', 'abc', '123');
   -- A1B2C3
   SELECT translate('12345', '14', 'ax');
   -- a23x5
   SELECT translate('', '14', 'ax');
   -- (empty string)
   SELECT translate('translate', 'rnlt', '123');
   -- 1a2s3ae
   SELECT translate('translate', '', '123');
   -- translate
   SELECT translate('translate', 'rnlt', '');
   -- asae
   SELECT translate('abcd', 'aba', '123');   -- first mapping of 'a' wins
   -- 12cd
   SELECT translate('abcd', 'aba', '12');
   -- 12cd
   SELECT translate('花花世界', '花界', 'ab');
   -- aa世b
   ```
   
   Supplementary code points (from `StringExpressionsSuite`, Java escapes):
   
   ```text
   translate("𠀋xyza𩹉c123b𠮷𩸽",
             "𩸽a𠮷b𠀋c𩹉c",
             "1🌻2🌷🐕🐈🌸")
   -> "🐕xyz🌻🌸🐈123🌷21"
   ```
   
   ### Edge cases to test
   
   - NULL in each of the three arguments returns NULL.
   - Empty `input`, empty `from` (no-op), and empty `to` (delete every `from` 
character).
   - Duplicate characters in `from`: the first mapping wins, including when the 
first one is a deletion. Using the same character twice in `from` with a longer 
`to` must not let the second mapping override the first.
   - A U+0000 character in `to` (e.g. `translate('abc', 'b', char(0))`) deletes 
`b` in Spark, giving `ac`. This was derived from `UTF8String.translate`, which 
skips values equal to `"\0"`.
   - Supplementary characters (emoji, CJK Extension B) in `input`, `from` and 
`to`, as in the suite case above. Combining marks are separate code points: 
`translate('é', 'e', 'x')` with a decomposed `é` changes only the base letter.
   - `from` and `to` as columns as well as literals. Spark caches the 
dictionary per distinct (`from`, `to`) pair, so the result must not depend on 
row order.
   - Non-string arguments are cast to STRING (e.g. `translate(12345, '14', 
'ax')`).
   - Collations (4.0+):
     - `UTF8_LCASE` matches characters case-insensitively. It uses a lowercased 
dictionary and treats `i` followed by U+0307 specially.
     - ICU collations use collation-aware matching.
     - `*_CS_AI` collations are rejected by the input type.
     - Non-`UTF8_BINARY` collations convert invalid UTF-8 with `toValidString`.
   
     Supporting `UTF8_BINARY` only and rejecting other collations is acceptable 
for a first version.
   
   ### Differences across Spark versions
   
   - 3.4 / 3.5: identical code-point semantics, `"\0"` deletion sentinel 
included.
   - 4.0: routes through `CollationSupport.StringTranslate`, which adds 
collation-aware variants. `UTF8_BINARY` behaviour is unchanged.
   - 4.1 / 4.2: none found.
   
   ### Implementation notes
   
   - Spark source: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
 (`StringTranslate`, `object StringTranslate.buildDict`). Kernel: 
`UTF8String.translate` in 
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`. 
Collation variants: `CollationAwareUTF8String.lowercaseTranslate` / `translate` 
in 
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java`.
   - DataFusion core `translate` 
(`datafusion/functions/src/unicode/translate.rs`) iterates over `char`s (code 
points) as of apache/datafusion#21405 (DataFusion 54). Older versions used 
graphemes. It already implements first-mapping-wins and deletes extra `from` 
characters. The remaining difference is the NUL case: a `to` character of 
U+0000 is written as NUL instead of deleting the input character. A thin Spark 
wrapper that post-processes or reimplements the dictionary build with the 
sentinel rule, reusing core's ASCII fast path, should be enough.
   - Module: `crates/spark/src/function/string/translate.rs`.
   - Tests: enable `crates/sqllogictest/test_files/string/translate.slt` and 
add the NUL, duplicate-key and supplementary-character cases. Document with 
`#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to