andygrove opened a new issue, #199:
URL: https://github.com/apache/datafusion-spark/issues/199
Part of #177 and apache/datafusion-comet#7300.
### What the function does
`translate(input, from, to)` replaces every character of `input` that
appears in `from` with the character at the same position in `to`. Available
since Spark 1.5.0.
- Characters are Unicode code points. Supplementary characters (surrogate
pairs in Java) are handled as single characters.
- If `from` is longer than `to`, the extra `from` characters are deleted
from `input`.
- If `to` is longer, its extra characters are ignored.
- If a character appears more than once in `from`, its first occurrence
decides the mapping.
- Spark builds the dictionary with U+0000 as the "delete" sentinel. As a
result, any `from` character explicitly mapped to U+0000 in `to` is also
deleted, not replaced with NUL.
All three arguments are STRING
(`StringTypeNonCSAICollation(supportsTrimCollation = true)`) via
`ImplicitCastInputTypes`, so numbers and similar values are cast to STRING. The
return type is `input`'s type. It is null-intolerant: NULL in any argument
returns NULL.
### Examples
```sql
SELECT translate('AaBbCc', 'abc', '123');
-- A1B2C3
SELECT translate('12345', '14', 'ax');
-- a23x5
SELECT translate('', '14', 'ax');
-- (empty string)
SELECT translate('translate', 'rnlt', '123');
-- 1a2s3ae
SELECT translate('translate', '', '123');
-- translate
SELECT translate('translate', 'rnlt', '');
-- asae
SELECT translate('abcd', 'aba', '123'); -- first mapping of 'a' wins
-- 12cd
SELECT translate('abcd', 'aba', '12');
-- 12cd
SELECT translate('花花世界', '花界', 'ab');
-- aa世b
```
Supplementary code points (from `StringExpressionsSuite`, Java escapes):
```text
translate("𠀋xyza𩹉c123b𠮷𩸽",
"𩸽a𠮷b𠀋c𩹉c",
"1🌻2🌷🐕🐈🌸")
-> "🐕xyz🌻🌸🐈123🌷21"
```
### Edge cases to test
- NULL in each of the three arguments returns NULL.
- Empty `input`, empty `from` (no-op), and empty `to` (delete every `from`
character).
- Duplicate characters in `from`: the first mapping wins, including when the
first one is a deletion. Using the same character twice in `from` with a longer
`to` must not let the second mapping override the first.
- A U+0000 character in `to` (e.g. `translate('abc', 'b', char(0))`) deletes
`b` in Spark, giving `ac`. This was derived from `UTF8String.translate`, which
skips values equal to `"\0"`.
- Supplementary characters (emoji, CJK Extension B) in `input`, `from` and
`to`, as in the suite case above. Combining marks are separate code points:
`translate('é', 'e', 'x')` with a decomposed `é` changes only the base letter.
- `from` and `to` as columns as well as literals. Spark caches the
dictionary per distinct (`from`, `to`) pair, so the result must not depend on
row order.
- Non-string arguments are cast to STRING (e.g. `translate(12345, '14',
'ax')`).
- Collations (4.0+):
- `UTF8_LCASE` matches characters case-insensitively. It uses a lowercased
dictionary and treats `i` followed by U+0307 specially.
- ICU collations use collation-aware matching.
- `*_CS_AI` collations are rejected by the input type.
- Non-`UTF8_BINARY` collations convert invalid UTF-8 with `toValidString`.
Supporting `UTF8_BINARY` only and rejecting other collations is acceptable
for a first version.
### Differences across Spark versions
- 3.4 / 3.5: identical code-point semantics, `"\0"` deletion sentinel
included.
- 4.0: routes through `CollationSupport.StringTranslate`, which adds
collation-aware variants. `UTF8_BINARY` behaviour is unchanged.
- 4.1 / 4.2: none found.
### Implementation notes
- Spark source:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
(`StringTranslate`, `object StringTranslate.buildDict`). Kernel:
`UTF8String.translate` in
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`.
Collation variants: `CollationAwareUTF8String.lowercaseTranslate` / `translate`
in
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java`.
- DataFusion core `translate`
(`datafusion/functions/src/unicode/translate.rs`) iterates over `char`s (code
points) as of apache/datafusion#21405 (DataFusion 54). Older versions used
graphemes. It already implements first-mapping-wins and deletes extra `from`
characters. The remaining difference is the NUL case: a `to` character of
U+0000 is written as NUL instead of deleting the input character. A thin Spark
wrapper that post-processes or reimplements the dictionary build with the
sentinel rule, reusing core's ASCII fast path, should be enough.
- Module: `crates/spark/src/function/string/translate.rs`.
- Tests: enable `crates/sqllogictest/test_files/string/translate.slt` and
add the NUL, duplicate-key and supplementary-character cases. Document with
`#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]