voonhous commented on code in PR #19414:
URL: https://github.com/apache/hudi/pull/19414#discussion_r3681516134
##########
hudi-io/src/test/java/org/apache/hudi/common/util/TestStringUtils.java:
##########
@@ -323,6 +323,66 @@ public void
testCompareUtf8BytesEmptyPrefixAndIdenticalStrings() {
assertEquals(0, StringUtils.compareUtf8Bytes("abc", "abc"));
}
+ @Test
+ public void testCompareUtf8BytesMatchesEncodedByteOrder() {
+ String[] alphabet = {
+ // One-byte UTF-8 characters, including the upper boundary.
+ "?",
+ "a",
+ String.valueOf((char) 0x007F),
+ // Two-byte UTF-8 lower and upper boundaries.
+ String.valueOf((char) 0x0080),
+ String.valueOf((char) 0x07FF),
+ // Three-byte UTF-8 boundaries around the surrogate range, plus U+FFFD.
+ String.valueOf((char) 0x0800),
+ String.valueOf((char) 0xD7FF),
+ String.valueOf((char) 0xE000),
+ String.valueOf((char) 0xFFFD),
+ // Four-byte UTF-8 supplementary characters, including two sharing a
high surrogate.
+ "😀", // U+1F600
+ new String(Character.toChars(0x20000)),
+ new String(Character.toChars(0x20001)),
+ new String(Character.toChars(0x10FFFF))
+ };
+
+ // Generate every sequence of one to three code points from the alphabet.
This covers cases
+ // where strings differ before, within, or after a supplementary character.
+ List<String> values = new ArrayList<>();
+ values.add("");
+ for (String first : alphabet) {
+ values.add(first);
+ for (String second : alphabet) {
+ values.add(first + second);
+ for (String third : alphabet) {
+ values.add(first + second + third);
+ }
+ }
+ }
+
+ // Compare only the sign because Comparator does not prescribe the
magnitude of its result.
+ for (String left : values) {
+ for (String right : values) {
+ assertEquals(
+ Integer.signum(compareEncodedUtf8Bytes(left, right)),
+ Integer.signum(StringUtils.compareUtf8Bytes(left, right)));
Review Comment:
Nit, feel free to ignore: when this 5.6M-pair loop fails it prints
`expected: <1> but was: <-1>` with no clue which pair failed, and most of these
operands are unprintable. A lazy message supplier only evaluates on failure:
```suggestion
assertEquals(
Integer.signum(compareEncodedUtf8Bytes(left, right)),
Integer.signum(StringUtils.compareUtf8Bytes(left, right)),
() -> "left=" + Arrays.toString(left.codePoints().toArray())
+ " right=" + Arrays.toString(right.codePoints().toArray()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
