yihua commented on PR #19898: URL: https://github.com/apache/hudi/pull/19898#issuecomment-6023717374
I traced when `listMatchingPartitionPaths` actually enumerates the table, since the description says a plain incremental pull always lists every partition. From reading master (not measured), the full listing is narrower than that. **No partition predicate** (`getAllQueryPartitionPaths` -> `BaseHoodieTableFileIndex.listPartitionPaths`): - Incremental query whose start is not `earliest` and is inside the active timeline: partitions come from the commit metadata of the instants in range (`TimelineUtils.getWrittenPartitions`, added in #10098). No table listing. The relation's `fileIndex` is built from `optParams`, so it takes this branch. - Start is `earliest` or older than the active timeline: every partition is listed. **With partition predicates** (prefix analysis is on by default): - Equality on all partition columns: one `exists` call, no listing. - Equality on the leading partition columns: listing under that prefix only. - No usable prefix: hive-style, URL-encoded tables call `getPartitionPathWithPathPrefixUsingFilterExpression` from the table root, which has no incremental shortcut. Other tables fall back to `getAllQueryPartitionPaths`, so the two cases above apply. So for `listFileSplits()`, the full-table listing this PR removes happens when: 1. the start is `earliest` or older than the active timeline, or 2. there is a partition predicate with no bound leading prefix on a hive-style, URL-encoded table. When the start is inside the active timeline and there is no such predicate, master already lists only the partitions written in the range, so the gain from the `listFileSplits()` change is small there. The prune rule goes through the same partition listing and then also loads the file slices of those partitions, so skipping it saves redundant work in every case, but it is only a full-table cost in the two cases above. Two asks: - @prashantwason which scenario were the "minutes of latency" measured in? Could the description say which cases list the full table instead of "always"? - Since this PR splits `listMatchingPartitionPaths` and adds `filterPartitionPaths` next to it, could the scaladoc of `listMatchingPartitionPaths` state when it lists the table (the cases above) and point callers that already hold candidate paths to `filterPartitionPaths`? Today it only says "Prune the partition by the filter". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
