yihua opened a new issue, #855: URL: https://github.com/apache/hudi-rs/issues/855
### Description of the bug When file groups are listed from the metadata table `files` partition, a file group that has only log files and no base file is dropped. `file_groups_from_files_partition_records` in `crates/core/src/file_group/builder.rs` builds file groups by iterating base files and attaching matching log files, so log files whose file id has no base file are discarded. Every record in such a file group is lost from the read. This is the shape a bucket index MOR table takes before its first compaction, and what Flink ingestion and any MOR insert that goes straight to a log file produce. The file system listing path already handles it: `list_file_groups_for_partition` builds a log-only file group from the leftover log files, and the file group reader reads a slice with no base file. So the result today depends on which listing path the table takes: a v8+ table with the metadata table enabled silently reads fewer rows than the same table with the metadata table disabled. Split out of #389, which covered the file system listing and the reader. ### Steps To Reproduce 1. Create a table version 8 MOR table with a bucket index and the metadata table enabled. 2. Upsert into it and do not run compaction, so each file group holds only log files. 3. Read the table with hudi-rs. ### Expected behavior The read returns the records in the log-only file groups, the same rows the file system listing path returns for the same table. ### Additional context The stale `TODO: support adding log files to file group without base files` on `FileGroup::add_log_file` in `crates/core/src/file_group/mod.rs` describes work that is already done and can go in the same change. The `table_log_only` fixture has no metadata table, so a regression test needs a fixture with one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
