dongxiaoman commented on a change in pull request #7243:
URL: https://github.com/apache/pinot/pull/7243#discussion_r686949991
##########
File path:
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/PartitionSegmentPruner.java
##########
@@ -163,48 +181,44 @@ public synchronized void refreshSegment(String segment) {
if (filterExpression == null) {
return segments;
}
- Set<String> selectedSegments = new HashSet<>();
- for (String segment : segments) {
- PartitionInfo partitionInfo = _partitionInfoMap.get(segment);
- if (partitionInfo == null || partitionInfo == INVALID_PARTITION_INFO
|| isPartitionMatch(filterExpression,
- partitionInfo)) {
- selectedSegments.add(segment);
- }
- }
- return selectedSegments;
+ return pruneSegments((partitionInfo, cachedPartitionFunction) ->
isPartitionMatch(filterExpression,
Review comment:
Updated the logic to do the following:
1. Changed signature to pass in a `SelectedSegments` class instead of a
plain Set<String>. That class has a precomputed Sha256 hash of sorted segments
so I will know that two sets are identical quickly;
2. If the two sets are identical, we will try to add
`INVALID_PARTITION_INFO` segments into them and also do other pruning properly
3. If the two sets are NOT identical, we will go through the old route. I
have observed it via logging that it happens a lot for real time segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
