hubgeter opened a new pull request, #70570: URL: https://github.com/apache/doris/pull/70570
### Background Iceberg equality deletes compare keys even when a query does not project those columns. Native readers must preserve the physical keys' comparison semantics through decoding, schema evolution and cached delete filters. ### What problem does this PR solve? Issue Number: None Related PR: #69711 (equality-delete comparison handling), #68934 (nanosecond type support) PR Type: Bugfix Problem Summary: A query that projects only an ordinary column can silently omit rows when a hidden equality-delete key is a zoned nanosecond timestamp. For example, data keys 100ns and 200ns both decode to microsecond zero, so a tombstone for 200ns also filters the 100ns row. Carry physical timestamp precision-loss metadata through Parquet/ORC schema discovery and cached delete filters. Return NotSupported when a present key needs a lossy non-NULL comparison. Empty deletes, NULL-only tombstones and NULL defaults for missing keys keep their safe behavior. ### Details - **Frequency:** Iceberg scans with applicable equality deletes whose physically stored keys lose zoned nanosecond precision, including queries that do not project the key. - **Root cause:** Physical type inference exposes microsecond carriers for zoned nanoseconds. Later comparison sees matching carrier types after the remainder has already been discarded. - **Why existing tests missed it:** Coverage included unzoned nanosecond comparison and missing zoned-nanosecond defaults, but did not cover a hidden key physically present in both files. - **Fix:** Concrete file readers expose a format-neutral precision-loss marker. The table reader checks both sides before adding a present-key predicate. Cached tombstone summaries retain NULL-only and missing-NULL behavior without rescanning the tombstone null map for every split. - **Other paths and branches:** Covers Parquet and ORC native readers, nested keys and timestamp mapping modes. branch-4.2 at 7523405f6aa8e9df3d110eb432fed1ff6e1c3418 is also affected and needs a semantic backport: it uses an older single comparison domain and different cache/default handling. - **Behavior change:** Unsupported non-NULL nanosecond key comparisons fail explicitly instead of returning incomplete results. This does not add zoned nanosecond SQL projection support. ### Testing - clang-format 16, check-format, build header hygiene and git diff whitespace checks passed. - Repository clang-tidy checks passed on changed lines in all six changed C++ files. Per-source analysis used the generated Unity compiler flags. - Added seven real-file reader tests covering Parquet/ORC data and delete combinations, both lossy sides, negative epochs, NULL/empty deletes, nested keys, cache reuse and an exact unzoned-nanosecond control. - Local ASAN validation is running through run-be-ut.sh with the equality-delete tests and the existing zoned-nanosecond missing-default test selected. The focused test executable uses a local CMake source selection that is excluded from this PR. No runtime test pass is claimed yet. - Live SQL and remote CI have not been verified. ### Release note Iceberg native scans now report an unsupported comparison when a non-NULL equality-delete key would lose zoned nanosecond precision, preventing silent omission of rows. Empty deletes, NULL-only tombstones and safe missing-key NULL defaults remain supported. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Not ready: local ASAN build and native reader test execution are still in progress. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, tests and this description. - Human review: Pending author review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
