uros-db commented on code in PR #46700:
URL: https://github.com/apache/spark/pull/46700#discussion_r1628820660
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -296,6 +344,45 @@ public static String toLowerCase(final String target,
final int collationId) {
return UCharacter.toLowerCase(locale, target);
}
+ /**
+ * Converts a single code point to lowercase using ICU rules, with special
handling for
+ * one-to-many case mappings (i.e. characters that map to multiple
characters in lowercase).
+ *
+ * @param codePoint The code point to convert to lowercase.
+ * @param sb The StringBuilder to append the lowercase character to.
+ */
+ private static void lowercaseCodePoint(final int codePoint, final
StringBuilder sb) {
+ if (codePoint == 0x0130) {
+ // Latin capital letter I with dot above is mapped to 2 lowercase
characters.
+ sb.appendCodePoint(0x0069);
+ sb.appendCodePoint(0x0307);
+ }
+ else if (codePoint == 0x03C2) {
+ // Greek final and non-final capital letter sigma should be mapped the
same.
+ sb.appendCodePoint(0x03C3);
+ }
+ else {
+ // All other characters should follow context-unaware ICU single-code
point case mapping.
+ sb.appendCodePoint(UCharacter.toLowerCase(codePoint));
Review Comment:
that's correct, since the logic of illegal UTF8 byte sequences is a bit
involved and requires thorough testing, I plan to do that in a follow-up PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
