andygrove opened a new issue, #203:
URL: https://github.com/apache/datafusion-spark/issues/203

   Part of #177 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `mask(input[, upperChar[, lowerChar[, digitChar[, otherChar]]]])` masks a 
string:
   
   - upper-case letters are replaced with `upperChar` (default `'X'`);
   - lower-case letters with `lowerChar` (default `'x'`);
   - decimal digits with `digitChar` (default `'n'`);
   - all other characters with `otherChar` (default NULL, which keeps them).
   
   Passing NULL for any replacement argument keeps the original characters of 
that class. Available since Spark 3.4.0. Since 3.5 it is registered through 
`MaskExpressionBuilder` and supports named arguments (`str`, `upperChar`, 
`lowerChar`, `digitChar`, `otherChar`).
   
   All five arguments must be STRING (`ExpectsInputTypes`, not implicit-cast, 
so `mask(123)` or an INT replacement argument is rejected with 
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`). The return type is `input`'s type. 
The four replacement arguments must be foldable (constant). Each one must be 
NULL or exactly one character (`UTF8String.numChars == 1`). Otherwise analysis 
fails with `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT` or 
`DATATYPE_MISMATCH.INPUT_SIZE_NOT_ONE`. The result is NULL only when `input` is 
NULL.
   
   The character class comes from Java `Character.getType(char)` on each UTF-16 
code unit: `UPPERCASE_LETTER`, `LOWERCASE_LETTER`, `DECIMAL_DIGIT_NUMBER`, or 
anything else.
   
   ### Examples
   
   ```sql
   SELECT mask('abcd-EFGH-8765-4321');
   -- xxxx-XXXX-nnnn-nnnn
   SELECT mask('abcd-EFGH-8765-4321', 'Q');
   -- xxxx-QQQQ-nnnn-nnnn
   SELECT mask('AbCD123-@$#', 'Q', 'q');
   -- QqQQnnn-@$#
   SELECT mask('AbCD123-@$#', 'Q', 'q', 'd', 'o');
   -- QqQQdddoooo
   SELECT mask('AbCD123-@$#', NULL, 'q', 'd', 'o');
   -- AqCDdddoooo
   SELECT mask('AbCD123-@$#', 'Q', 'q', NULL, 'o');
   -- QqQQ123oooo
   SELECT mask('abcd-EFGH-8765-4321', NULL, NULL, NULL, '*');
   -- abcd*EFGH*8765*4321
   SELECT mask('AbCD123-@$#', NULL, NULL, NULL, NULL);
   -- AbCD123-@$#
   SELECT mask(NULL, NULL, NULL, NULL, 'o');
   -- NULL
   SELECT mask('AbCD123-@$#', lowerChar => 'q', upperChar => 'Q', digitChar => 
'd');
   -- QqQQddd-@$#
   SELECT mask('AbCD123-@$#', 'Qa', 'qa', 'da', 'oa');
   -- error: DATATYPE_MISMATCH.INPUT_SIZE_NOT_ONE (exprName = upperChar)
   ```
   
   ### Edge cases to test
   
   - NULL `input` returns NULL whatever the other arguments are. A NULL 
replacement argument keeps that class unchanged.
   - Each replacement argument, one at a time, is NULL while the others are set.
   - A replacement argument longer than one character, or an empty string 
(`numChars == 0`), fails with `DATATYPE_MISMATCH.INPUT_SIZE_NOT_ONE`. A column 
replacement argument fails with `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT`. In 
DataFusion terms, reject non-scalar replacement arguments at planning time.
   - Non-ASCII letters and digits follow Unicode general categories:
     - `É` and `Ω` are upper case and `ß` is lower case;
     - CJK ideographs are `OTHER_LETTER`, so they count as "other";
     - titlecase letters such as `Dž` count as "other";
     - non-ASCII decimal digits such as Arabic-Indic `٣` are digits.
   - Supplementary characters are classified **per UTF-16 code unit**. Each 
half of a surrogate pair is `Character.SURROGATE`, so it counts as "other":
     - with `otherChar => 'o'`, one emoji (or `𝐀`, U+1D400, which is an 
uppercase letter as a code point) becomes `oo`;
     - with `otherChar` NULL, it is kept unchanged.
   
     Iterating Rust `char`s would classify U+1D400 as upper case and produce a 
single replacement, which does not match Spark. This was derived from 
`Mask.transformInput` (`input.toString.map { ... }`).
   - A replacement argument that is a supplementary character (for example 
`'😀'`) passes the `numChars == 1` check, but only `charAt(0)` (the high 
surrogate) is written. `UTF8String.fromString` then encodes the lone surrogate 
as `?`, so `mask('A', '😀')` gives `?`. This was derived from the code; Spark 
has no test for it.
   - Input with an explicit NULL `otherChar` versus an omitted one (both keep 
the character).
   - Collations (4.0+): any collation is accepted for all arguments and the 
result keeps `input`'s collation. Masking ignores collation.
   
   ### Differences across Spark versions
   
   - 3.4: `Mask` is registered directly. The documented positional forms work; 
there are no named arguments.
   - 3.5: `MaskExpressionBuilder` adds named-argument support.
   - 4.0: collation-aware input types (`StringTypeWithCollation`). The default 
literals are created with `Literal.create(..., StringType)`, and the NULL 
default `otherChar` takes `input.dataType`. Semantics are unchanged.
   - 4.1 / 4.2: none found.
   
   ### Implementation notes
   
   - Spark source: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala`
 (`MaskExpressionBuilder`, `Mask`, `object Mask.transformInput` / 
`transformChar`).
   - DataFusion core has no `mask` function, so this needs a new kernel. 
Iterate UTF-16 code units, or emulate them: treat every supplementary code 
point as two "other" units. Classify BMP characters with Java's 
`Character.getType` categories (Unicode general categories Lu, Ll, Nd). The 
Unicode version of the JDK and of Rust's tables may differ slightly for newly 
assigned characters.
   - Module: `crates/spark/src/function/string/mask.rs`. Named-argument support 
depends on DataFusion's named-argument support for UDFs. Positional forms are 
the priority.
   - Tests: `crates/sqllogictest/test_files/string/mask.slt` exists as a 
placeholder with commented-out queries. Enable them and add the Unicode and 
surrogate cases. Document with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to