mkaravel commented on code in PR #46700: URL: https://github.com/apache/spark/pull/46700#discussion_r1628799978
########## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java: ########## @@ -296,6 +344,45 @@ public static String toLowerCase(final String target, final int collationId) { return UCharacter.toLowerCase(locale, target); } + /** + * Converts a single code point to lowercase using ICU rules, with special handling for + * one-to-many case mappings (i.e. characters that map to multiple characters in lowercase). + * + * @param codePoint The code point to convert to lowercase. + * @param sb The StringBuilder to append the lowercase character to. + */ + private static void lowercaseCodePoint(final int codePoint, final StringBuilder sb) { + if (codePoint == 0x0130) { + // Latin capital letter I with dot above is mapped to 2 lowercase characters. + sb.appendCodePoint(0x0069); + sb.appendCodePoint(0x0307); + } + else if (codePoint == 0x03C2) { + // Greek final and non-final capital letter sigma should be mapped the same. + sb.appendCodePoint(0x03C3); + } + else { + // All other characters should follow context-unaware ICU single-code point case mapping. + sb.appendCodePoint(UCharacter.toLowerCase(codePoint)); Review Comment: I believe this does not map illegal UTF8 byte sequences to U+FFFD. Do we plan to make this change as part of this PR, or in a follow up one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org