morningman opened a new pull request, #70226:
URL: https://github.com/apache/doris/pull/70226
### Background
An Iceberg equality delete removes the data rows whose key columns equal one
of its rows, its tombstones. A table can add a column after some data files
were written and then use it as a key; those files don't store it, so Iceberg
reads it there as the column's initial default, or NULL when it has none, and
NULL equals only a NULL tombstone. The BE's V2 Iceberg reader builds such a
missing key as a literal from the schema metadata that the FE sends with the
scan, and matches a data file without field ids, written before `migrate` or
`add_files`, through the table's name mapping.
### What problem does this PR solve?
Issue Number: None
Related PR: #70093 (an earlier version of these tests, built on old-FE
plans; closed)
PR Type: Test
Problem Summary:
A missing key without an initial default must read as NULL and match NULL
tombstones only; master tests that on Parquet alone, with a single key, almost
always in files with field ids. This PR adds two matrices, for a top-level and
a nested missing key, on the scan plan the current FE sends, across ORC and
Parquet, files with and without field ids, and composite and single-key
deletes. It also fixes the ORC fixture writer, which stored child values under
NULL parents and so shifted later rows. Only test code changes.
### Details
- **Change map:** 338 added lines, all in
`be/test/format_v2/table/iceberg_reader_test.cpp`: the two matrices (218),
helpers that build the current FE's scan plan and name mappings (41), a writer
for the top-level key's delete files (73), and the ORC fixture fix (6).
- **Scenarios:** `IcebergMissingEqualityKeyMatchesOnlyNullTombstones`, 24
cases: a file with `id` = 1, 2, 3, written before the table added the optional
key `added_key` (INT or STRING, no initial default), in Parquet or ORC, with or
without field ids. The delete (id, added_key) = (2, NULL), (3, 0 or '') removes
row 2 only; a single NULL tombstone removes every row, with a predicate of NULL
literals alone; a single 0 or '' tombstone removes nothing.
`IcebergMissingNestedEqualityKeyMatchesOnlyNullTombstones`, 8 cases: the key
`payload.k`, added after a file that stores `id` and a nullable `payload
{existing}`, NULL in row 2, in Parquet or ORC, with or without field ids, the
query reading `id` only or `payload` too. The tombstones (1, NULL payload) and
(3, 0) leave rows 2 and 3, and a projected `payload` keeps its values.
- **What existing tests miss:** The closest tests use the missing key as the
only key, against a NULL tombstone on Parquet only
(`IcebergEqualityDeleteMatchesNullForMissingDataColumn`,
`IcebergEqualityDeleteMatchesNullMissingKeyUnderUnknownTableType`) or against
tombstones far from a NULL slot's value, such as 7
(`IcebergEqualityDeleteMissingKeyDoesNotReadUnsupportedUnprojectedCarrier`,
`IcebergMissingWholeStructEqualityKeyMaterializesDefault`). None matches a NULL
tombstone in ORC, uses a STRING key, pairs the missing key with a present one,
or deletes with a tombstone equal to a NULL slot's value; a key missing under a
present ancestor is tested only with a non-NULL default
(`IcebergMissingNestedEqualityKeyPreservesNullableParent`).
- **Fails when broken:** Two mutations, each run against all of
`IcebergV2ReaderTest`. Reading a missing key without a default as its type's
default, 0 or '', instead of NULL (`build_missing_equality_delete_key_expr`)
fails all 32 new cases. Letting a NULL data key also match, and hash as, the
value its nested column holds (`EqualityDeletePredicate`) fails the 24 new
cases with a non-NULL tombstone, while the 8 single NULL-tombstone cases pass,
as they should. The two mutations also fail 3 and 5 existing tests.
### Testing
- `run-be-ut.sh --run
--filter='IcebergV2ReaderTest.*:IcebergTableReaderTest.*'`, locally on macOS
(arm64, ASAN build) with `be/src/format_v2` and `be/test/format_v2` as on this
PR's head: 129 tests pass, the 2 new ones included.
### Notes for Reviewers
- `write_nested_equality_orc_file` now marks a child slot NULL under a NULL
parent. The ORC writer stores a child's PRESENT bits only for rows whose parent
is present but its values for every row the child batch marks present, so
`existing` = {10, 20, 30} under a NULL row 2 read back as 10, NULL, 20. The
projected-`payload` case exposed it; the helper's other callers never read a
child value after a NULL parent and pass unchanged.
- The cases without field ids carry an authoritative name mapping on every
field, as `migrate` and `add_files` set it and the current FE sends it: under
scan semantics v2 such a file is read by name only through one
(`IcebergTableReader::_mapping_mode`).
### Release note
None
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: the tests, the fixture fix, and this description
- Human review: not stated
- Prompts: Rebuild the two missing-key NULL matrices of #70093 on current
scan semantics, with complete schema metadata for the keys and their ancestors
and name mappings for files without field ids, as the author's comment closing
#70093 asked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
