yabola commented on code in PR #1023:
URL: https://github.com/apache/parquet-mr/pull/1023#discussion_r1113905243
##########
parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java:
##########
@@ -289,8 +320,14 @@ public <T extends Comparable<T>> Boolean visit(Lt<T> lt) {
T value = lt.getValue();
- // drop if value <= min
- return stats.compareMinToValue(value) >= 0;
+ // we are looking for records where v < someValue
+ if (stats.compareMinToValue(value) >= 0) {
+ // drop if value <= min
+ return BLOCK_CANNOT_MATCH;
+ } else {
+ // if value > min, we must take it
+ return BLOCK_MUST_MATCH;
Review Comment:
> Dictionary contains the very values from the pages themselves
@gszadovszky do you think it is suitable to use `BLOCK_MUST_MATCH` when all
page are encoded in `DictionaryFilter`?
For example,
1. when search for data = x1 and dictionary has one element = x1 , then
return `BLOCK_MUST_MATCH`
2. when search for data > x1 and every elements in dictionary > x1 , then
return `BLOCK_MUST_MATCH`
##########
parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java:
##########
@@ -289,8 +320,14 @@ public <T extends Comparable<T>> Boolean visit(Lt<T> lt) {
T value = lt.getValue();
- // drop if value <= min
- return stats.compareMinToValue(value) >= 0;
+ // we are looking for records where v < someValue
+ if (stats.compareMinToValue(value) >= 0) {
+ // drop if value <= min
+ return BLOCK_CANNOT_MATCH;
+ } else {
+ // if value > min, we must take it
+ return BLOCK_MUST_MATCH;
Review Comment:
> Dictionary contains the very values from the pages themselves
@gszadovszky do you think it is suitable to use `BLOCK_MUST_MATCH` when all
page are encoded in `DictionaryFilter`?
For example,
1. when search for data = x1 and dictionary has one element = x1 , then
return `BLOCK_MUST_MATCH`
2. when search for data > x1 and every elements in dictionary > x1 , then
return `BLOCK_MUST_MATCH`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
