cshuo commented on code in PR #20544:
URL: https://github.com/apache/hudi/pull/20544#discussion_r4522640691
##########
hudi-common/src/main/java/org/apache/hudi/common/avro/AvroRecordContext.java:
##########
@@ -107,6 +119,36 @@ public Object getValue(IndexedRecord record, HoodieSchema
schema, String fieldNa
return getFieldValueFromIndexedRecord(record, fieldName);
}
+ @Override
+ public Comparable getOrderingValue(IndexedRecord record, HoodieSchema
schema, List<String> orderingFieldNames) {
+ return OrderingValues.create(orderingFieldNames, field -> {
+ Object value = getOrderingValueFromIndexedRecord(record, field,
consistentLogicalTimestampEnabled);
+ return value == null ? OrderingValues.getDefault() :
ensureComparability(value);
+ });
+ }
+
+ @Override
+ public Comparable getOrderingValue(IndexedRecord record, HoodieSchema
schema, String[] orderingFieldNames) {
+ return orderingFieldNames == null
+ ? OrderingValues.getDefault()
+ : OrderingValues.create(orderingFieldNames, field -> {
+ Object value = getOrderingValueFromIndexedRecord(record, field,
consistentLogicalTimestampEnabled);
+ return value == null ? OrderingValues.getDefault() :
ensureComparability(value);
+ });
+ }
+
+ /**
+ * Extracts an ordering field from a generic Avro record, normalizing its
logical type.
+ *
+ * @param record the record to read; must be a {@link GenericRecord}
+ * @param fieldName the field name, which may be a nested path
+ * @param consistentLogicalTimestampEnabled whether to use consistent
logical timestamp conversion
+ * @return the ordering value, or null if the field or its value is missing
+ */
+ public static Object getOrderingValueFromIndexedRecord(IndexedRecord record,
String fieldName, boolean consistentLogicalTimestampEnabled) {
Review Comment:
The distinction is between **reading a field for reuse in a record** and
**extracting an ordering value for comparison**:
| Accessor | Behavior | Example: DATE |
|---|---|---|
| `getFieldValueFromIndexedRecord` | Returns the value unchanged |
`Integer(1)` |
| `getNestedFieldVal` | Normalizes logical types, with parameters
controlling missing-field handling and timestamp representation |
`LocalDate(1970-01-02)` |
- **Why keep ordinary field access unchanged?**
Partial updates use `RecordContext.getValue` and put the extracted values
directly into a reconstructed Avro record. With consistent logical timestamps
enabled, converting a `timestamp-micros` value from `Long` to
`java.sql.Timestamp` here would cause serialization to fail: the writer expects
a `long`, but receives a `Timestamp`.
- **Why normalize ordering values?**
The merger compares `BufferedRecord`s constructed from either
`HoodieRecord`s or engine records. Different Avro versions and reader
configurations can also produce different value representations, such as
`Integer` versus `LocalDate`. Since the merger calls `compareTo` directly, both
paths must produce the same comparable representation.
- **What this PR changes:**
It aligns `AvroRecordContext#getOrderingValue` with
`HoodieAvroIndexedRecord#doGetOrderingValue`, which already used
`getNestedFieldVal`. Both use `returnNullIfNotFound=true` and the same
effective `consistentLogicalTimestampEnabled` setting. Ordinary field access
remains unchanged: `AvroRecordContext#getValue` and
`HoodieAvroIndexedRecord#getColumnValues` both continue to call
`getFieldValueFromIndexedRecord`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]