[
https://issues.apache.org/jira/browse/AVRO-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julio J. Gomez Diaz updated AVRO-4354:
--------------------------------------
Attachment: avro-jira-repro.zip
> [java] Generated equals() (AVRO-3527) throws "Unknown datum type" for arrays
> containing logical-type unions, and generated hashCode() is inconsistent with
> equals() for CharSequence fields
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AVRO-4354
> URL: https://issues.apache.org/jira/browse/AVRO-4354
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.12.1, 1.12.2
> Environment: h2. Environment
> Java 25 (Temurin 25.0.4), Maven 3.9.16, {{avro-maven-plugin}} with its
> default configuration, JUnit 5.11.4.
> Reporter: Julio J. Gomez Diaz
> Priority: Major
> Attachments: avro-jira-repro.zip
>
>
> h2. Summary
> Since 1.12.1 the {{SpecificCompiler}} generates {{equals()}} and
> {{hashCode()}} for every record (AVRO-3527). Compared to 1.12.0, where
> records inherited {{SpecificRecordBase.equals()}} / {{{}hashCode(){}}}, this
> introduces two regressions:
> # *{{equals()}} throws* {{AvroRuntimeException: Unknown datum type ...}}
> when the record has an array whose elements contain a union of {{null}} and a
> logical type with a registered conversion ({{{}uuid{}}}, {{{}date{}}},
> {{{}timestamp-millis{}}}, ...). It happens both for an _array of records with
> a nullable logical-type field_ and for an {_}array of nullable logical-type
> values{_}. It is also asymmetric: {{before.equals(after)}} returns
> {{{}true{}}}, while {{after.equals(before)}} throws.
> # *{{hashCode()}} is inconsistent with {{equals()}}* for {{CharSequence}}
> fields (the default {{{}stringType{}}}): a record holding a
> {{java.lang.String}} and the same record after a round-trip, which holds an
> {{{}org.apache.avro.util.Utf8{}}}, are equal according to {{equals()}} but
> have different hash codes. This breaks {{HashSet}} / {{HashMap}} lookups.
> Serialization and deserialization are *not* affected. The problems appear
> when generated objects are compared or hashed: test assertions,
> deduplication/idempotency with {{Set}} / {{{}Map{}}},
> {{{}List.contains(){}}}, {{{}Stream.distinct(){}}}, argument matching in
> mocking libraries, etc. For duplicates, both decoded objects have the same
> hash code, so {{HashSet.add()}} / {{contains()}} and {{HashMap.get()}} always
> end up calling {{equals()}} and always throw.
> h2. Affected versions
> * *1.12.1* and {*}1.12.2{*}: reproduced (see the results table).
> * {{branch-1.12}} (1.12.3-SNAPSHOT) and {{{}main{}}}: not executed. Code
> inspection shows the same generated code in {{record.vm}}
> ({{{}java.util.Objects.equals(this.x, other.x){}}} and {{{}x.hashCode(){}}})
> and the same {{{}GenericData.AbstractArray.equals(){}}}.
> * {*}1.12.0{*}: not affected, because no {{equals()}} / {{hashCode()}} is
> generated.
> * Classes generated with the 1.12.0 compiler and run on the 1.12.1 runtime
> are *not* affected. The regression is in the generated code, not in the
> runtime.
> h2. How to reproduce
> Plain {{avro-maven-plugin}} with its default configuration (only the
> {{schema}} goal, {{stringType}} left at its default {{{}CharSequence{}}}):
> {code:xml}
> <plugin>
> <groupId>org.apache.avro</groupId>
> <artifactId>avro-maven-plugin</artifactId>
> <version>1.12.1</version>
> <executions>
> <execution>
> <phase>generate-sources</phase>
> <goals>
> <goal>schema</goal>
> </goals>
> </execution>
> </executions>
> </plugin>
> {code}
> Schema ({{{}src/main/avro/order.avsc{}}}):
> {code:json}
> [
> {
> "type": "record",
> "name": "Order",
> "namespace": "org.example",
> "fields": [
> {
> "name": "lines",
> "type": {
> "type": "array",
> "items": {
> "type": "record",
> "name": "Line",
> "fields": [
> { "name": "line_id", "type": ["null", { "type": "string",
> "logicalType": "uuid" }], "default": null }
> ]
> }
> },
> "default": []
> },
> {
> "name": "dates",
> "type": { "type": "array", "items": ["null", { "type": "int",
> "logicalType": "date" }] },
> "default": []
> }
> ]
> },
> {
> "type": "record",
> "name": "Named",
> "namespace": "org.example",
> "fields": [
> { "name": "name", "type": "string" }
> ]
> }
> ]
> {code}
> Test (JUnit 5):
> {code:java}
> package org.example;
> import static org.junit.jupiter.api.Assertions.assertEquals;
> import static org.junit.jupiter.api.Assertions.assertTrue;
> import java.io.IOException;
> import java.time.LocalDate;
> import java.util.HashSet;
> import java.util.List;
> import java.util.Set;
> import java.util.UUID;
> import org.junit.jupiter.api.Test;
> class GeneratedEqualsTest {
> private static final UUID ID =
> UUID.fromString("de5714a7-0000-0000-0000-000000000001");
> private static Order roundTrip(final Order order) throws IOException {
> return Order.getDecoder().decode(Order.getEncoder().encode(order));
> }
> @Test
> void arrayOfRecordsWithNullableUuid() throws IOException {
> final Order before = Order.newBuilder()
> .setLines(List.of(Line.newBuilder().setLineId(ID).build()))
> .setDates(List.of())
> .build();
> final Order after = roundTrip(before);
> assertTrue(before.equals(after)); // passes on 1.12.0 and
> 1.12.1
> assertTrue(after.equals(before)); // 1.12.1:
> AvroRuntimeException: Unknown datum type java.util.UUID
> assertTrue(after.equals(roundTrip(before))); // 1.12.1: same exception
> }
> @Test
> void arrayOfNullableDate() throws IOException {
> final Order before = Order.newBuilder()
> .setLines(List.of())
> .setDates(List.of(LocalDate.of(2026, 9, 28)))
> .build();
> final Order after = roundTrip(before);
> assertTrue(after.equals(before)); // 1.12.1:
> AvroRuntimeException: Unknown datum type java.time.LocalDate
> }
> @Test
> void hashCodeIsConsistentWithEquals() throws IOException {
> final Named before = Named.newBuilder().setName("a").build();
> // holds a java.lang.String
> final Named after =
> Named.getDecoder().decode(Named.getEncoder().encode(before)); // holds an
> org.apache.avro.util.Utf8
> assertTrue(before.equals(after)); // true on 1.12.0 and
> 1.12.1
> assertEquals(before.hashCode(), after.hashCode()); // 1.12.1: fails,
> equal objects with different hash codes
> final Set<Named> set = new HashSet<>(Set.of(before));
> assertTrue(set.contains(after)); // 1.12.1: false
> }
> }
> {code}
> h3. Results
> ||Test||1.12.0||1.12.1||1.12.2 (note 1)||Generator 1.12.0 + runtime 1.12.1||
> |arrayOfRecordsWithNullableUuid|pass|{{AvroRuntimeException: Unknown datum
> type java.util.UUID}}|same as 1.12.1|pass|
> |arrayOfNullableDate|pass|{{AvroRuntimeException: Unknown datum type
> java.time.LocalDate}}|same as 1.12.1|pass|
> |hashCodeIsConsistentWithEquals|pass|{{AssertionFailedError: expected: <128>
> but was: <159>}}|same as 1.12.1|pass|
> Note 1: on 1.12.2 the test has to run with
> {{{}-Dorg.apache.avro.SERIALIZABLE_PACKAGES=org.example{}}}. Without it,
> every decode fails earlier with {{{}SecurityException: Forbidden
> org.example.Order! This class is not trusted to be included in Avro schemas
> ...{}}}, even when using the generated {{{}getDecoder(){}}}.
> Stack trace on 1.12.1:
> {noformat}
> org.apache.avro.AvroRuntimeException: Unknown datum type java.util.UUID:
> de5714a7-0000-0000-0000-000000000001
> at
> org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:976)
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:935)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:1309)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:1285)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:1299)
> at
> org.apache.avro.generic.GenericData$AbstractArray.equals(GenericData.java:358)
> at org.example.Order.equals(Order.java:373)
> at
> org.example.GeneratedEqualsTest.arrayOfRecordsWithNullableUuid(GeneratedEqualsTest.java:32)
> {noformat}
> where {{Order.java:373}} is the generated line:
> {code:java}
> if (!java.util.Objects.equals(this.lines, other.lines)) {
> {code}
> h2. Root cause
> h3. 1. The equals() exception
> The generated {{equals()}} compares non-primitive, non-{{{}CharSequence{}}}
> fields with {{java.util.Objects.equals(this.x, other.x)}} (the
> {{canGenerateEqualsAndHashCode}} block of {{{}record.vm{}}}). On a decoded
> record, the runtime type of an array field is {{{}GenericData.Array{}}},
> whose {{equals()}} is:
> {code:java}
> public boolean equals(final Object o) {
> if (!(o instanceof Collection)) {
> return false;
> }
> return GenericData.get().compare(this, o, this.getSchema(), true) == 0;
> }
> {code}
> {{GenericData.get()}} has no logical-type conversions registered. When
> {{compare()}} reaches the union, {{resolveUnion()}} calls
> {{{}getSchemaName(){}}}, which cannot classify the {{java.util.UUID}} /
> {{java.time.LocalDate}} / {{java.time.Instant}} value and throws.
> This defect of {{GenericData.AbstractArray.equals()}} already exists in
> 1.12.0. AVRO-4036 reports it for a direct {{List.equals()}} call with a
> custom logical type. In 1.12.0 it could only be reached by calling
> {{equals()}} on the list directly: {{SpecificRecordBase.equals()}} uses
> {{{}getSpecificData(){}}}, i.e. the {{MODEL$}} of the generated class, which
> carries the conversions, to compare the whole tree. Since AVRO-3527, *every*
> generated {{equals()}} of a record containing such an array goes through
> {{{}GenericData.Array.equals(){}}}.
> The asymmetry comes from the list implementation on each side:
> * a {{java.util.List}} built by the user compares element by element:
> generated {{{}Line.equals(){}}}, then {{{}UUID.equals(){}}};
> * a {{GenericData.Array}} produced by the decoder uses
> {{{}GenericData.get(){}}}.
> h3. 2. The hashCode() inconsistency
> The generated {{hashCode()}} calls {{x.hashCode()}} directly, while the
> generated {{equals()}} uses {{Utf8.compareSequences()}} for {{CharSequence}}
> fields. {{String.hashCode()}} and {{Utf8.hashCode()}} differ ({{{}"a"{}}}: 97
> vs 128), so two objects that are equal according to {{equals()}} get
> different hash codes. In 1.12.0, {{SpecificRecordBase.hashCode()}} normalised
> strings through {{GenericData.hashCode()}} ({{{}new
> Utf8(o.toString()).hashCode(){}}}), so {{equals()}} and {{hashCode()}} were
> consistent. The same happens with strings inside collections; see AVRO-4198
> for the {{equals()}} side.
> h3. 3. No way to opt out
> {{SpecificCompiler.canGenerateEqualsAndHashCode(Schema)}} only returns
> {{false}} when custom logical type factories are used, so no compiler or
> {{avro-maven-plugin}} option can restore the previous behaviour. The only
> workaround is a custom {{templateDirectory}} with a copy of {{record.vm}}
> that lacks the equals/hashCode block. That copy has to be kept in sync with
> every release, because the templates also carry security fixes (e.g.
> AVRO-4053 / CVE-2025-33042), and because the compiler runs Velocity in strict
> mode, so templates from one version fail with the compiler of another.
> h2. Suggested fix
> # Generated {{{}equals(){}}}: for non-primitive fields (at least arrays,
> maps and unions), delegate to the class model instead of
> {{{}java.util.Objects.equals(){}}}, e.g. {{{}MODEL$.compare(this.x, other.x,
> SCHEMA$.getFields().get(i).schema(), true) == 0{}}}, or generate an
> element-wise comparison. Primitive and {{CharSequence}} fields can keep the
> fast path introduced by AVRO-3527.
> # Additionally, or alternatively, make
> {{GenericData.AbstractArray.equals()}} use the {{GenericData}} instance that
> created the array (the one that called {{{}newArray(){}}}) instead of
> {{{}GenericData.get(){}}}. This would also fix AVRO-4036.
> # Generated {{{}hashCode(){}}}: hash {{CharSequence}} values independently
> of their implementation, as {{GenericData.hashCode()}} does, or delegate
> non-primitive fields to {{{}MODEL$.hashCode(value, schema){}}}, so that the
> result stays consistent with {{{}equals(){}}}.
> # Add a compiler / {{avro-maven-plugin}} option (e.g.
> {{{}createEqualsAndHashCode{}}}) to disable the generation of {{equals()}} /
> {{{}hashCode(){}}}, so users can fall back to {{SpecificRecordBase}}
> semantics without forking the templates.
> h2. Related issues
> * AVRO-3527: introduced the generated {{equals()}} / {{hashCode()}} (cause).
> * AVRO-4036: {{GenericData.Array.equals()}} with logical-type unions (same
> underlying defect, direct call, 1.12.0).
> * AVRO-4198: the generated {{equals()}} returns {{false}} for {{String}} vs
> {{Utf8}} in complex types after a round-trip.
> * AVRO-4334: {{Utf8.hashCode()}} values changed in 1.12.1.
> * AVRO-4183, AVRO-4188: other regressions of the generated methods (fields
> named {{result}} / {{{}java{}}}).
> * AVRO-4139: equality of arrays of maps, fixed in 1.12.1 (different code
> path).
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
