zhongyujiang commented on code in PR #1023:
URL: https://github.com/apache/parquet-mr/pull/1023#discussion_r1113749755
##########
parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java:
##########
@@ -289,8 +320,14 @@ public <T extends Comparable<T>> Boolean visit(Lt<T> lt) {
T value = lt.getValue();
- // drop if value <= min
- return stats.compareMinToValue(value) >= 0;
+ // we are looking for records where v < someValue
+ if (stats.compareMinToValue(value) >= 0) {
+ // drop if value <= min
+ return BLOCK_CANNOT_MATCH;
+ } else {
+ // if value > min, we must take it
+ return BLOCK_MUST_MATCH;
Review Comment:
> From the code, minMax and dictionary is basically the minMax of real
data. But the specification doesn't say.
IIUC, there may be other implementation libraries where min-max is not part
of the row-group data set, because the specification does not define it
clearly. Although I don't know of any library that does this.
> Can we add a config (default false, don't use BLOCK_MUST_MATCH) to
optimize it?
From my point of view, it is doable to introduce such a config, but I am a
little concerned that it will be difficult for users to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
