[
https://issues.apache.org/jira/browse/AVRO-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-4267:
---------------------------------
Labels: pull-request-available (was: )
> GenericData.compare(...) orders bytes using signed ByteBuffer.compareTo,
> unlike binary Avro order
> -------------------------------------------------------------------------------------------------
>
> Key: AVRO-4267
> URL: https://issues.apache.org/jira/browse/AVRO-4267
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Reporter: Ruiqi Dong
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Summary*
> `BinaryData.compare(...)` compares Avro `bytes` using unsigned byte order.
> `GenericData.compare(...)` falls through to `ByteBuffer.compareTo(...)` for
> `bytes`, which uses signed Java byte comparison. This makes in-memory datum
> comparison inconsistent with binary-encoded datum comparison.
>
> *Affected code*
> File: `lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java`
> {code:java}
> protected int compare(Object o1, Object o2, Schema s, boolean equals) {
> ...
> case STRING:
> CharSequence cs1 = o1 instanceof CharSequence ? (CharSequence) o1 :
> o1.toString();
> CharSequence cs2 = o2 instanceof CharSequence ? (CharSequence) o2 :
> o2.toString();
> return Utf8.compareSequences(cs1, cs2);
> default:
> return ((Comparable) o1).compareTo(o2);
> }
> } {code}
> File: `lang/java/avro/src/main/java/org/apache/avro/io/BinaryData.java`
> {code:java}
> public static int compareBytes(byte[] b1, int s1, int l1, byte[] b2, int s2,
> int l2) {
> return Arrays.compareUnsigned(b1, s1, s1 + l1, b2, s2, s2 + l2);
> } {code}
> *Reproducer*
> Add this test to
> `lang/java/avro/src/test/java/org/apache/avro/TestCompare.java`
> {code:java}
> @Test
> void genericBytesCompareMatchesBinaryUnsignedByteOrder() throws Exception {
> Schema schema = SchemaParser.parseSingle("\"bytes\"");
> ByteBuffer lower = ByteBuffer.wrap(new byte[] { 0x7F });
> ByteBuffer higher = ByteBuffer.wrap(new byte[] { (byte) 0xFF });
> assertTrue(BinaryData.compare(render(lower, schema, new
> GenericDatumWriter<>()), 0,
> render(higher, schema, new GenericDatumWriter<>()), 0, schema) < 0);
> assertTrue(GenericData.get().compare(lower, higher, schema) < 0);
> } {code}
> Run:
> {code:java}
> MAVEN_SKIP_RC=true
> JAVA_HOME=/opt/homebrew/Cellar/openjdk@21/21.0.6/libexec/openjdk.jdk/Contents/Home
> \
> PATH=/opt/homebrew/Cellar/openjdk@21/21.0.6/libexec/openjdk.jdk/Contents/Home/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin
> \
> /opt/homebrew/bin/mvn -q -t toolchains-local.xml -pl lang/java/avro \
>
> -Dtest=org.apache.avro.TestCompare#genericBytesCompareMatchesBinaryUnsignedByteOrder
> test {code}
> *Observed behavior*
> The test fails on the `GenericData.compare(...)` assertion. Binary comparison
> orders `0x7F < 0xFF`, but generic in-memory comparison orders the same
> buffers the other way because `(byte) 0xFF` is `-1` in Java.
> *Expected behavior*
> `GenericData.compare(...)` should use the same byte ordering as
> `BinaryData.compare(...)` for Avro `bytes`.
> `BinaryData.compare(...)` is documented as consistent with
> `GenericData.compare(...)`. Different ordering for the same logical `bytes`
> values can affect sorting, comparisons inside records, and any code that
> expects binary and in-memory Avro datum ordering to agree.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
