linliu-code opened a new pull request, #693:
URL: https://github.com/apache/hudi-rs/pull/693
## Description
**Stacked on #692, which is stacked on #691 and #689. Review those first.**
Until they merge the diff here shows their commits too; the delta is
`4e1d4a3..HEAD`.
The metadata table states `hoodie.record.merge.mode=CUSTOM`, and version two
refuses CUSTOM, so a metadata file slice with log files cannot be read through
it. Five places refused: the schema handler, the record merger factory, the
record buffer loader, the merge-mode resolver (twice, and `MergeMode` had no
way to represent CUSTOM at all since it is a two-variant enum), and the file
group reader's own version gate. All five now resolve through one function, so
they admit or refuse together.
Selection is on the payload class, not the merge strategy id. The metadata
table's id is the all-zeros value Java names
`PAYLOAD_BASED_MERGE_STRATEGY_UUID`, which every payload-based custom table
carries and which means only that the payload decides; selecting on it would
claim tables this merger has no business merging. A CUSTOM table naming any
other payload keeps erroring rather than being merged by an ordering rule it
did not ask for, including one that names a real merger of its own under the
pre-version-eight spelling of the strategy id.
The merger mirrors `HoodieMetadataPayload.preCombine`. All six partition
types share one flat Arrow schema with an integer type discriminator, so this
is one merger dispatching on one column, and the six rules collapse to three
shapes. Bloom filters, record index and secondary index take the newer record
whole, which leaves the payload a zero-copy row reference. Files and
all-partitions fold the `filesystemMetadata` map. Column statistics and
partition statistics share a rule that widens the bounds and sums four
counters. A folded record is the newer row with one column replaced, matching
Java, which carries every other field from the newer payload.
The vectorized merge kernel needed a change of its own. It decides
base-versus-log conflicts inline through `pick_winner`, which models the two
ordering modes and nothing else, so a mode it cannot express fell into the
event-time branch, found no ordering value on either side, and took the log
record whole. That would have dropped every file listed only in the base file,
silently, on any slice with both a base file and log files. A mode
`pick_winner` does not model now goes through the merger, as the row path
already did; the two ordering modes keep the allocation-free fast path.
Two limits, stated rather than left to be discovered. A statistics bound
whose union branch differs between the two records is refused rather than
reconciled: Java promotes across wrapper types using a `valueType` field the
schemas here do not carry, so promotion could be written but not tested, and
column statistics drive file pruning, where a bound reported under the wrong
type prunes away files holding matching rows. A folded statistics record also
holds its source block's union buffers alive, because slicing a dense union
keeps its children whole; that is retention, not corruption, and it is worth
measuring before optimising.
Nothing routes the metadata table through version two yet: the version gate
sends it to version one before config is read, and the adapter refuses it. This
makes version two capable of the merge and changes no read. The layer that
would use it is separate work.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Three reads of real metadata slices, each with an oracle that does not come
from this code. A `files` slice, base file plus seven log files, is compared
entry by entry against what `MetadataTableFileGroupReader` returns for the same
slice, which folds through a separate implementation on decoded structs rather
than Arrow; before this change the same read returns four keys holding one file
entry each where the fold holds fourteen. A `partition_stats` slice, eight log
files and no base file, is compared against the newest record per key, built by
reading each log file alone in commit order. A `secondary_index` base file plus
a log file that deletes its record is asserted to return nothing, with a guard
that the base file held a record to begin with.
Base-versus-log folding is covered separately, through `merge_base_batch`
rather than through the metadata fixtures. The fixtures cannot cover it: their
base file holds one record whose entries are a subset of what the log fold
already produces, so they pass whether or not the base row is folded. That was
found by mutating the kernel's call site rather than the fold, which is now
part of the routine below.
Every rule was mutation-checked, and each mutation fails the test that names
that rule: the fold skipped, the fold's call site removed, min and max swapped,
`min` for `max` on file size, a tombstone marking instead of cancelling, a
tombstone losing to an older tombstone, the two statistics short-circuits
removed, a null bound winning, the selection types folding, a delete not
winning, and the version gate's narrowing reverted.
Five rules have no real-data coverage and are covered by unit tests alone,
which is a property of the fixture rather than an omission: `column_stats`
record keys never repeat, since the key is per partition, file and column and
each commit writes new files; every `partition_stats` record in the fixture is
tight-bound, so that rule short-circuits on all ninety-nine of them; and no
index key exists as live data on both sides, so both index slices merge by
deleting. `bloom_filters` has no fixture at all. A fixture that produces those
situations is separate work.
Run locally on both legs: default features and `--no-default-features`,
clippy clean on both with warnings denied, fmt clean, and `cargo check
--workspace --all-targets --all-features` clean including the Python and C++
bindings. No CI run has happened: fork pull requests sit at `action_required`
until a committer approves them, so all of the above is local only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
