wangzhigang1999 opened a new issue, #10492: URL: https://github.com/apache/paimon/issues/10492
### Search before asking I searched existing issues for NaN and floating-point primary keys. [#8532](https://github.com/apache/paimon/issues/8532) covers Java generated record equality; this report concerns the Python primary-key writer and merge reader. ### Paimon version PyPaimon from master at `5d902e8e047f4e2e8fb1341647d52e21bb12761b` (2.2-SNAPSHOT). ### Compute Engine Python API, Python 3.12.6 and PyArrow 19.0.1. The reproduction uses Parquet, one bucket, and Python planning/reading/writing/committing. ### Minimal reproduce step Write two distinct DOUBLE primary keys, `1.0` and `NaN`, in separate commits, then read the table: ```python import math import tempfile import pyarrow as pa from pypaimon import CatalogFactory, Schema with tempfile.TemporaryDirectory() as warehouse: catalog = CatalogFactory.create({'warehouse': warehouse}) catalog.create_database('default', True) arrow_schema = pa.schema([('id', pa.float64()), ('value', pa.string())]) catalog.create_table('default.nan_key', Schema.from_pyarrow_schema( arrow_schema, primary_keys=['id'], options={ 'bucket': '1', 'file.format': 'parquet', 'scan.native-plan.enabled': 'false', 'read.native.enabled': 'false', 'write.native.enabled': 'false', 'commit.native.enabled': 'false', }), False) table = catalog.get_table('default.nan_key') for row in [{'id': 1.0, 'value': 'finite'}, {'id': float('nan'), 'value': 'nan'}]: builder = table.new_batch_write_builder() writer, committer = builder.new_write(), builder.new_commit() try: writer.write_arrow(pa.Table.from_pylist([row], schema=arrow_schema)) committer.commit(writer.prepare_commit()) finally: writer.close() committer.close() rb = table.new_read_builder() splits = rb.new_scan().plan().splits() actual = rb.new_read().to_arrow(splits).to_pylist() print('files:', sum(len(split.files) for split in splits)) print('raw_convertible:', [split.raw_convertible for split in splits]) print('actual:', [dict(row, id='NaN' if math.isnan(row['id']) else row['id']) for row in actual]) assert sorted(row['value'] for row in actual) == ['finite', 'nan'], 'finite primary key was dropped' ``` On the baseline, both files contain their input rows, but the merged read returns: ```text files: 2 raw_convertible: [False] actual: [{'id': 'NaN', 'value': 'nan'}] AssertionError: finite primary key was dropped ``` The same reproduction fails with `pa.float32()`. ### What doesn't meet your expectations? The reader should return both rows. Java's floating-point key comparator distinguishes a finite number from NaN, treats NaNs as equal for the same remaining key fields, and orders `-0.0` before `+0.0` as distinct keys. The Python primary-key comparator uses `<` and `>` to infer equality. Both return false for a finite number versus NaN, so the merge reader treats unrelated keys as equal and keeps only one version. The file-interval comparator has the same NaN problem for composite keys. The writer also needs the same ordering and equality rules: Arrow's floating sort places NaNs first with the current null-placement option and treats signed zeros as ties, while Python tuple equality folds the two zeros together and can leave multiple NaN versions in one file. Changing only the reader would violate its sorted-run assumptions for these files. ### Anything else? I propose aligning the Python writer's sort/fold and the reader's key/interval comparisons with Java's FLOAT/DOUBLE semantics. Existing Python files may have incompatible ordering or already-folded signed-zero rows; handling those files needs a separate rewrite or recovery path. The proposed fix covers the semantics of new writes. This report uses one bucket and full scans. NaN predicate statistics and routing different NaN payloads across buckets need separate investigation. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
