alitrack opened a new pull request, #969:
URL: https://github.com/apache/hudi-rs/pull/969
## Description
`file_groups_from_files_partition_records` (the metadata-table `files`
partition path) built file groups by iterating base files and attaching
matching log files, so log files whose file id had no base file were
silently discarded. Every record in such a file group was lost from the
read.
That is the shape a bucket-index MOR table takes before its first
compaction, and what Flink ingestion and any MOR insert that goes
straight to a log file produce. The file system listing path
(`list_file_groups_for_partition`) already builds a log-only file group
from its leftover log files, so a v8+ table with the metadata table
enabled read fewer rows than the same table with it disabled.
This change builds a file group from the leftover log files, mirroring
the storage-listing path. `FileGroup::add_log_file` already places log
files with no base file into log-only slices, so no reader changes were
needed.
Also:
- replaces `test_log_files_without_base_file_not_included`, which
asserted the dropping behavior, with a test for the log-only group
- adds a test mixing a normal group and a log-only group in one
partition
- removes the two stale `TODO` comments that described log-only file
groups as not yet supported (the reader support landed with #389)
Closes #855
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
