Guosmilesmile commented on PR #17280: URL: https://github.com/apache/iceberg/pull/17280#issuecomment-5102450799
@swapna267 > So this is individual Iceberg Scan request for each lookup key right Yes, correct . In PARTIAL mode each lookup(keyRow) call issues its own TableScan.filter(col = v).planTasks() and opens the resulting data files. There is no cross-key batching today. > Would a merging layer help? In principle yes, but it's not free, which is why I kept it out of this PR. It's not free, which is why I kept it out of this PR. * Adds buffering latency (a max-batch-size / max-batch-latency knob), which changes lookup-join semantics slightly. Interacts non-trivially with Flink's built-in LookupCache (partial cache) . We need to split the batch into "cache hits" vs. "cache misses" before issuing the scan, and re-associate results with driving rows. * Large IN lists have their own cost in Iceberg planning: InPredicate evaluates min/max against every literal per manifest entry, so the win vs. N separate scans depends on N, key selectivity, and how well the table is clustered on the lookup column. For a poorly-clustered table, one big scan can end up reading almost everything and be worse than N small pruned scans. * Iceberg's Expressions.in also has an internal threshold (IN_PREDICATE_LIMIT, default 200) above which it degrades. > For the current PR's scope Small-to-medium dimension tables that are well-clustered on the lookup key, or covered by FULL cache, per-key scans are acceptable. I'd propose we land the current single-key implementation first, and then further improve this feature in the following PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
