JunRuiLee opened a new pull request, #8075:
URL: https://github.com/apache/paimon/pull/8075
### Purpose
The pypaimon merge read path broke key-ties using only the file-level
sequence
number (`HeapEntry.__lt__`), ignoring the user-configured `sequence.field`.
For
PK tables that set it, this silently returned the wrong "latest" value
whenever
write/file order disagreed with the sequence-field order.
Mirrors Java's `SortMergeReaderWithMinHeap` + `UserDefinedSeqComparator`:
- **sort_merge_reader**: add `builtin_seq_comparator` so heap order becomes
key → sequence.field → file sequence number. Handles multiple fields,
descending sort order, and parameterized types (`TIMESTAMP(p)`,
`DECIMAL(p,s)`).
- **table_read**: re-inject and project out a sequence field dropped by a
projection.
- **aggregation_merge_function**: don't aggregate the sequence column (force
`last_value`), matching Java `getAggFuncName`.
- **merge_engine_support**: add `check_sequence_field_valid` mirroring Java
`SchemaValidation.validateSequenceField` (pypaimon has no schema-creation
validation) — rejects a missing/duplicated sequence field, an aggregator on
the sequence column, `sequence.field` on FIRST_ROW, and cross-partition
update. These raise `ValueError`.
### Tests
`test_sequence_field_read.py` + additions to `test_aggregation_e2e.py`:
covers
deduplicate / partial-update / aggregation ordering, multiple fields,
descending order, parameterized types, projection dropping the field, and all
rejection cases. Unset `sequence.field` keeps the existing
file-sequence-number
fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
