chrevanthreddy opened a new issue, #20164: URL: https://github.com/apache/hudi/issues/20164
**What needs to be done:** `HoodieBackedTableMetadata#getRecordsByKeyPrefixes` currently sends every encoded prefix to every file slice in the requested metadata partition. That fallback is correct for metadata partitions whose records are distributed by full-key hashing, because a prefix does not identify one target file group. `MetadataPartitionType.VECTOR_INDEX` has a stronger routing contract. Its write path uses `HoodieTableMetadataUtil#mapVectorPostingKeyToFileGroupIndex`, which routes posting records by IVF cluster ID. A posting lookup prefix such as `P|<generation>|<cluster>|<shard>|` therefore identifies the same target MDT file group as every posting row matching that prefix. Add a vector-specific read-side routing path that: 1. Detects a `VECTOR_INDEX` posting-prefix lookup whose encoded prefixes contain the generation and cluster fields required by the vector routing contract. 2. Maps each encoded prefix with the same `mapVectorPostingKeyToFileGroupIndex` function used by the write path. 3. Groups prefixes by target file-group index. 4. Scans only the corresponding latest merged file slices, passing each slice only its routed prefixes. 5. Falls back to the existing all-file-slices behavior for non-vector partitions, non-posting/empty/unknown/malformed prefixes, or any mixed request that cannot be routed safely. Keep this change in the existing metadata-table prefix-read seam. Do not add a second vector-only HFile reader or duplicate file-slice resolution. **Why this task is needed:** RFC-109 candidate generation calls `getRecordsByKeyPrefixes` for the selected IVF posting prefixes. The current generic implementation scans every vector MDT file group for every request even though vector writes already establish deterministic routing. For the agreed 10M layout of roughly 50–100 vector MDT file groups, a query probing 32 clusters currently schedules reads against all 50–100 file slices even when those clusters route to a smaller set. The historical 4096-file-group diagnostic made this fanout catastrophic; the corrected 50–100-group layout reduces the magnitude but does not remove the unnecessary HFile open/seek and task fanout. The performance gain at the corrected layout must be measured rather than assumed. This is a follow-up performance optimization, not a correctness prerequisite for completing RFC-109. The vector read path must remain correct with the existing full-fanout metadata prefix reader; this task reduces unnecessary MDT file-slice work and can be picked up independently. This is not ordinary Spark table-partition predicate pruning. It is metadata-table file-slice pruning for routed vector key-prefix lookups. ## Verified current behavior - `MetadataPartitionType.VECTOR_INDEX#getFileGroupMappingFunction` delegates to `mapVectorPostingKeyToFileGroupIndex`. - Posting routing uses the IVF cluster ID, preserving one target file group for all shards/rows in a cluster. - `HoodieBackedTableMetadata#getRecordsByKeyPrefixes` currently loads all latest merged slices and scans each slice with the complete prefix list. - `VectorPostingReader` reaches that generic API through `metadataTable.getRecordsByKeyPrefixes(...)`. - An earlier diagnostic implementation (`c1fffe304c`) demonstrated this routing shape, but it is not part of the current RFC-109 change chain and should be reworked/tested against current master rather than copied blindly. ## Acceptance criteria - Results are identical to the existing full-fanout prefix lookup for routeable vector keys. - A routeable vector prefix is evaluated only against its mapped file slice. - Multiple prefixes mapped to one file group are scanned together. - Non-vector partitions retain existing behavior. - Unrouteable, non-posting, or mixed vector prefix requests conservatively retain existing behavior; they must never over-prune. - Prefix ordering remains compatible with UTF-8 HFile key ordering within each target slice. - Tests cover one and multiple file groups, multiple prefixes sharing a group, prefixes spanning groups, malformed/unsupported prefixes, and non-vector fallback. - Diagnostics or tests make the selected-slice count observable so the optimization cannot silently regress to all-slice fanout. # Task Type Performance optimization # Related Issues **Parent feature issue:** #19094 — RFC-109: MDT-backed vector index umbrella tracking issue **Related issues:** #19101 (query planning and posting-range derivation), #19102 (posting scan and approximate candidates), and #18852 (write-side file-group routing). This is a non-blocking follow-up that can be implemented independently through the existing metadata posting-read seam. It is also independent of optional HFile block-cache enhancements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
