wangzhigang1999 opened a new issue, #10433: URL: https://github.com/apache/paimon/issues/10433
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master at `23e6f84c9be3d76e750fca545bfc845729cc7299`. ### Compute Engine PyPaimon with Python 3.12.6 and PyArrow 19.0.1. The reproducer uses Python write, commit, scan planning and read paths. ### Minimal reproduce step Configure `rowkind.field=op` on a primary-key table, write a row with `op='-D'`, commit it, and reopen the table: ```python from tempfile import TemporaryDirectory import pyarrow as pa from pypaimon import CatalogFactory, Schema with TemporaryDirectory() as warehouse: catalog = CatalogFactory.create({'warehouse': warehouse}) catalog.create_database('default', True) arrow_schema = pa.schema([ pa.field('id', pa.int32(), nullable=False), pa.field('v', pa.int32()), pa.field('op', pa.string()), ]) options = { 'bucket': '1', 'file.format': 'parquet', 'rowkind.field': 'op', 'write.native.enabled': 'false', 'commit.native.enabled': 'false', 'scan.native-plan.enabled': 'false', 'read.native.enabled': 'false', } catalog.create_table('default.events', Schema.from_pyarrow_schema( arrow_schema, primary_keys=['id'], options=options), False) table = catalog.get_table('default.events') builder = table.new_batch_write_builder() writer, committer = builder.new_write(), builder.new_commit() try: writer.write_arrow(pa.Table.from_pylist( [{'id': 1, 'v': 10, 'op': '-D'}], schema=arrow_schema)) committer.commit(writer.prepare_commit()) finally: writer.close() committer.close() table = CatalogFactory.create({'warehouse': warehouse}).get_table('default.events') reader = table.new_read_builder() splits = reader.new_scan().plan().splits() print('delete counts:', [f.delete_row_count for s in splits for f in s.files]) print('raw convertible:', [s.raw_convertible for s in splits]) print('rows:', reader.new_read().to_arrow(splits).to_pylist()) ``` Actual output: ```text delete counts: [0] raw convertible: [True] rows: [{'id': 1, 'v': 10, 'op': '-D'}] ``` ### What doesn't meet your expectations? A retract-only key should be absent from a snapshot read. Expected output: ```text delete counts: [1] raw convertible: [False] rows: [] ``` The Python writer persists the DELETE row with `_VALUE_KIND=3`, but records `delete_row_count=0` in its file metadata. The planner therefore treats this single-file split as raw-convertible, and the reader bypasses merge and retract filtering. UPDATE_BEFORE (`-U`) has the same problem. This also reproduces with ORC and with a point lookup that isolates a retract-only file after file rolling. ### Anything else? The common writer also records zero delete counts for input changelog files. Counts should describe the physical DELETE and UPDATE_BEFORE rows in each output file after buffer merging and file rolling. Related work: [#6735](https://github.com/apache/paimon/issues/6735) tracks RowKind support, and [#10368](https://github.com/apache/paimon/pull/10368) added retract counting for postpone writes. This report concerns the remaining common-writer metadata path. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
