xiangfu0 commented on PR #16776: URL: https://github.com/apache/pinot/pull/16776#issuecomment-4665167099
Closing this after re-checking the routing/pruning paths. The smoking gun is that broker partition pruning is already per segment, not based on a single table-wide partition count. `SinglePartitionColumnSegmentPruner` looks up each segment's own `SegmentPartitionInfo` from ZK and evaluates the filter against that segment's partition function/count. If the metadata is missing or invalid, the pruner keeps the segment instead of pruning it, so it is conservative and does not drop rows. That means Kafka topic expansion does not by itself create a query-correctness issue for normal broker pruning. Old segments can carry the old partition count and new segments can carry the expanded partition count; each segment is evaluated against its own metadata. The remaining value of this change is only for table-level partition metadata consumers, such as partition-aware routing / MSQ worker assignment. There, remapping is valid only when the expanded partition count is an exact multiple of the configured logical partition count, e.g. `4 -> 8`, because `hash % 8 % 4 == hash % 4`. It is not a general topic-expansion correctness fix, and the consuming-segment fallback cannot verify the total expanded partition count from the LLC segment name alone. Given that, this PR is not needed as a correctness fix and is too narrow/ambiguous to carry as a new table config knob. Closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
