alitrack opened a new pull request, #819: URL: https://github.com/apache/hudi-rs/pull/819
## Why All Hudi indexes live in the metadata table (`<table>/.hoodie/metadata`), partitioned by index type. The only public Arrow read today is for the `files` partition (`Table::read_metadata_table_files_partition_arrow`); the other partitions are reachable only through the private `MetadataTable::partition_reader` (used by tests). Downstream engines that want to surface Hudi's index data (the DuckDB extension is the motivating case: https://github.com/alitrack/duckdb-hudi) cannot read `record_index` / `bloom_filters` / `column_stats` / `partition_stats` / `secondary_index_*` through the public API. ## What - Generalise `MetadataTableV2Reader::read_files_partition_batch` into `read_partition_batch(partition)` — the read is identical, only the partition name passed to `rebuild_record_context` / `InputSplit` changes. `read_files_partition` stays as a thin wrapper. - `MetadataTable::read_files_partition_batch` wraps a new `read_metadata_table_partition_batch(partition, keys, valid_instants)`. - New **public** `Table::read_metadata_table_partition_arrow(partition, keys)` on DATA tables, mirroring the existing files-partition API. Returns an empty batch when the partition is not enabled or has no commits. ## Behaviour - `files` partition: byte-identical output to before (same reader path, same partition name). - Other partitions: raw Arrow batch in the partition's stored schema (union-of-records; the MDT record is an Avro union keyed by the `type` column). Callers consume the batch directly — there is no decoded record type for these partitions in this crate. ## Tests - Existing metadata-table tests exercise `files` through the generalised path unchanged. (Add a unit test reading `record_index` on the existing multi-shard fixture once CI is green — the fixture `metadata_bloom_filters_slice` and the v9 MDT test tables are available.) ## Note for reviewers Unrelated pre-existing issue found while verifying: `crates/test` requires `zip = "8"` (workspace), and every 8.x release of `zip` is currently yanked (9.0.0–9.0.2 too; only 9.0.3+ resolves), so a clean checkout of main does not resolve dependencies today. Verified this PR by locally bumping the workspace `zip` to `9.0.3` (not part of this diff). Happy to send a separate `build(deps): zip 8 → 9.0.3` PR if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
