linliu-code opened a new pull request, #692:
URL: https://github.com/apache/hudi-rs/pull/692
## Description
**Stacked on #691, which is stacked on #689. Review those first.** Until
they merge the diff here shows their commits too; the delta is `fd712c9..HEAD`,
three files.
A file slice holding an HFile log block reads as though the slice had no log
blocks: the read succeeds and returns the base file's rows. Nothing errors, and
no counter records a skipped block, so the loss is silent. Version two is the
default reader, and the version one fallback does not cover this because
version two never reports an inability, it reports success.
The block type was skipped at five points: the prefetch window planner, pass
one's classification into the per-instant map, pass three's decode dispatch,
pass three's buffer dispatch, and the buffer itself, whose `if let
LogBlockContent::Records(..)` has no else and discards anything other shape.
Admitting the type at the first four alone would not have helped, because the
buffer takes Arrow and nothing else.
An HFile block's records are Avro values stored under HFile keys, so they
are decoded against the schema in the block header, which is the same
resolution an Avro block already gets, and the buffer receives batches like
every other data block type. That resolution now lives in one helper shared by
both paths so they cannot drift on promotions. Whether an HFile block is
decoded or handed back as key-value pairs is the caller's choice, because the
metadata table reader wants the pairs.
A writer may leave the record's key field empty because the HFile entry key
already holds it, so an empty field is filled from the entry key. This shares
one implementation with the base file reader added in #691, including its
guards: a populated value and a null value are both left alone.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`harness_hfile_log_block_rejected` asserted `Invalid block type: 4`, which
held upstream where HFile blocks were unrecognised, and was ignored with a note
that the expectation needed re-deriving rather than inverting. It is re-derived
and un-ignored as `harness_hfile_log_block_merges`.
Its expectation comes from the inputs, not from this reader: the base file
was read with the parquet reader, and the block's two records were decoded with
`apache_avro::from_avro_datum` against the schema in the block header. Both
sides carry `ts = 100`, so event-time ordering ties and the case's commit-time
ordering decides, which makes the log write win on two of the four keys.
Without the fix the read returns the base file unchanged, so the case fails on
the commit time of the first row.
The `key` column is asserted because it is where the entry key is filled.
The base file holds it and the log writer left it empty, so without the fill
two rows read as empty next to two holding their own keys.
This is **not** a Hudi parity oracle. The fixture has no gold data, because
Spark cannot read it either, so nothing here checks hudi-rs against Hudi; it
checks the merge against the decoded inputs.
Run locally: 1337 pass with default features, 1317 with
`--no-default-features`, clippy clean on both feature sets with warnings
denied, fmt clean, and the metadata table reader's own test still passes, which
is what confirms that path still receives raw key-value pairs. No CI run has
happened: fork pull requests sit at `action_required` until a committer
approves them, so all of the above is local only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
