SEPURI-SAI-KRISHNA commented on PR #57922: URL: https://github.com/apache/spark/pull/57922#issuecomment-5573922947
@uros-b good question. I checked, and these paths are collation-independent. I have added a test that asserts it rather than just asserting it here. `substring`, `lpad` and `rpad` are not collation-aware functions. `CollationSupport` defines collation-aware implementations for 17 operations (`Contains`, `StartsWith`, `EndsWith`, `Upper`, `Lower`, `InitCap`, `FindInSet`, `StringInstr`, `StringReplace`, `StringLocate`, `SubstringIndex`, `StringTranslate`, the three `StringTrim*`, `StringSplitSQL` and `StringInstrWithOccurrence`), and none of them is a substring or a pad. Concretely: - `Substring.nullSafeEval` branches on `StringType` vs `BinaryType` only, never on the collation, and then calls `substringSQL`. - `StringLPad.nullSafeEval` and `StringRPad.nullSafeEval` call `UTF8String.lpad` and `UTF8String.rpad` unconditionally. There is one implementation for every collation. The collation reaches these expressions only through `dataType = str.dataType`, which propagates the input collation to the result type, and this PR does not touch typing. That is not accidental: collation governs comparison, ordering and case folding, while these functions do character position arithmetic (`numChars`, byte offsets, `copyOfRange`), which no collation changes. Most of this PR is in `ByteArray.java`, which serves `BinaryType`, where collation does not apply at all. I have added `SPARK-58708: LPAD/RPAD length handling is collation-independent` to `StringExpressionsSuite`. It runs the fixed non-positive lengths, plus ordinary positive lengths, under UTF8_BINARY, UTF8_LCASE, UNICODE and UNICODE_CI and asserts identical results. The mixed-case expectations also confirm that nothing is case folded under the case-insensitive collations. One nuance worth flagging: `inputTypes` uses `StringTypeWithCollation(supportsTrimCollation = true)`, so RTRIM collations are accepted, and under those the result of `rpad` compares equal to the unpadded input. That is pre-existing comparison semantics rather than padding behavior, and this PR does not change it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
