gszadovszky commented on code in PR #1023:
URL: https://github.com/apache/parquet-mr/pull/1023#discussion_r1113966806
##########
parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java:
##########
@@ -289,8 +320,14 @@ public <T extends Comparable<T>> Boolean visit(Lt<T> lt) {
T value = lt.getValue();
- // drop if value <= min
- return stats.compareMinToValue(value) >= 0;
+ // we are looking for records where v < someValue
+ if (stats.compareMinToValue(value) >= 0) {
+ // drop if value <= min
+ return BLOCK_CANNOT_MATCH;
+ } else {
+ // if value > min, we must take it
+ return BLOCK_MUST_MATCH;
Review Comment:
@yabola, I think you misunderstand how dictionary/filtering works. The
dictionary contains all of the values which the dictionary encoded pages may
contain. These pages actually not contain the values but the indices
referencing to the related values in the dictionary.
So, if a searched element can be found in the dictionary you may return
`BLOCK_MUST_MATCH` even if only one page is dictionary encoded. For example if
the filter is `x > 1` then any element in the dictionary `> 1` would fulfill
the filter so the `BLOCK_MUST_MATCH`.
If the dictionary does not contain any of the searched elements (for the
previous example every elements are `<= 1`) then you may return
`BLOCK_CANNOT_MATCH` only if all the related pages are dictionary encoded.
Otherwise you return `BLOCK_MIGHT_MATCH` since you don't know anything about
the not dictionary encoded pages based on the dictionary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
