wangzhigang1999 opened a new pull request, #10493:
URL: https://github.com/apache/paimon/pull/10493
Fixes #10492
### Purpose
PyPaimon can merge a finite FLOAT/DOUBLE primary key with NaN and drop an
unrelated row. For example, after separate commits of `(1.0, 'finite')` and
`(NaN, 'nan')`, a merged read returns only the NaN row. Writer folding also
collapses `-0.0` and `+0.0` into one key and can retain multiple versions of a
NaN key in one file.
This PR applies Java floating-point ordering to Python primary-key merging,
file-interval comparisons, and writer sorting/folding. NaNs compare equal above
numeric values; negative and positive zero remain distinct keys. Nulls stay
first, and composite keys continue comparing the remaining fields after equal
floating fields.
The writer derives canonical sort columns from the stored `_KEY_` fields and
uses the same columns for fold equality. It preserves the stored floating-point
values. The shared comparator accepts Python and NumPy floating scalars, and
non-floating read fields keep inline comparisons. Existing sequence-field
ordering remains unchanged.
#### Compatibility
The reader requires input files sorted in Java primary-key order with one
row per key. Older Python files can violate that prerequisite: NaNs sorted
before finite numbers, and signed zeros were sort ties. For example, an old
composite-key file containing `(+0.0, 'a')` before `(-0.0, 'b')` is descending
under the corrected comparator.
This PR fixes new writes; it does not reorder, reject, or migrate historical
files. Affected files require a separate rewrite before relying on the
corrected read order. Recovering rows already collapsed by the old writer
requires source data or another retained history. NaN predicate-statistics
pruning and cross-bucket NaN payload routing remain outside this change.
### Tests
- Added 10 regression cases covering FLOAT/DOUBLE real-table writes,
same-file NaN folding, independent signed-zero updates, nullable composite
keys, physical Parquet order/min-max, NaN file intervals, and Python/NumPy
scalar representations.
- Related local regression runs: 141
comparator/write-buffer/interval/sequence/sort-merge tests and 59
reader/merge-engine tests passed. Two Lance cases were excluded because the
local optional dependency is unavailable. After the latest writer edits, the 28
floating-key/write-buffer tests passed again.
- An additional 24 Python table scenarios passed against independently
specified key identities and latest-version expectations.
- Supplemental public-API Java/Python interoperability: 12 new Parquet
tables, 24 snapshots, and 48 reads passed. Java used jars built from the same
master baseline. Coverage includes both writer directions, mixed-language
updates, FLOAT/DOUBLE special values and composite keys, physical
ordering/bounds, and a NaN upper-bound file followed by a finite-key update.
These checks used bucket=1 with Python native backends disabled.
- Ruff, repository-configured Flake8, and `git diff --check` passed. Pyright
reported zero diagnostics in the two new files and all added/modified code
lines; seven existing diagnostics remain in unchanged code and match master.
Full CI and native backend lanes have not run.
A local comparator-sorting microbenchmark measured about 1.44 times the
baseline elapsed time for 20k finite DOUBLE keys, with an integer control of
about 1.01 times. This measures comparator sorting; end-to-end read throughput
was not measured.
Run the regression cases with:
```bash
PYTHONPATH=paimon-python python -m pytest -q \
paimon-python/pypaimon/tests/test_floating_primary_key.py \
paimon-python/pypaimon/tests/test_write_merge_buffer.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
