dongxiaoman commented on a change in pull request #7243:
URL: https://github.com/apache/pinot/pull/7243#discussion_r687023731
##########
File path:
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerFactory.java
##########
@@ -84,6 +82,8 @@ private SegmentPrunerFactory() {
}
}
}
+ // Always prune out empty segments first
Review comment:
> For the partition pruning, since in most cases a table will use the
same partition function for all segments, we can group the segments in a
different way:
>
> * Map<PartitionFunction, Map<Integer, Set>>: partition function ->
partition id -> segments
>
> When getting a query, we can loop over the partition functions from this
map and calculate all the matching segments. Then we loop over the input
segments and keep the segments that does not have partition info, or included
in the matching segments.
Good suggestion; the major challenge of this approach would be analyzing the
`isPartitionMatch(Expression filterExpression, PartitionInfo partitionInfo)`
method to pick out the segments properly. In a SQL filter we could have
something like `WHERE account_key in ['ac23d', 'e92df'] OR account_key =
'c3f9d'`, without proper rewrite it will be hard to make our recursive
selection right.
For now my code has become a targeted optimization mostly for Offline table
queries, because the externalView is much more stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
