[
https://issues.apache.org/jira/browse/PARQUET-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692207#comment-17692207
]
ASF GitHub Bot commented on PARQUET-2237:
-----------------------------------------
yabola commented on code in PR #1023:
URL: https://github.com/apache/parquet-mr/pull/1023#discussion_r1114365987
##########
parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java:
##########
@@ -289,8 +320,14 @@ public <T extends Comparable<T>> Boolean visit(Lt<T> lt) {
T value = lt.getValue();
- // drop if value <= min
- return stats.compareMinToValue(value) >= 0;
+ // we are looking for records where v < someValue
+ if (stats.compareMinToValue(value) >= 0) {
+ // drop if value <= min
+ return BLOCK_CANNOT_MATCH;
+ } else {
+ // if value > min, we must take it
+ return BLOCK_MUST_MATCH;
Review Comment:
@gszadovszky @wgtmac @zhongyujiang Thank you very much for working on it. I
have some thoughts.
> We can improve (?) the case when not all the pages are dictionary encoded
1. I can't make sure if it is suitable to load dictionary even if pages are
not all decoded. (I may choose not to change this behavior)
2. However considering the origin `BloomFilter` bug in parquet v1, we might
have to do something to avoid using `BloomFilter`(even if pages are all
encoded).
In the code implementation we may have to use some flag to mark if
dictionary `DictionaryFilter#expandDictionary` successfully (method will throw
`IOException` and we can't `expandDictionary` again in `BloomFilterImpl`).
Or we could also use `BLOCK_MUST_MATCH` like this PR.
> StatisticsFilter: Because of the lower/upper bound issue we cannot really
improve this (except for the specific case when min=max)
If we only use it when min=max, I think it might not really improve .
> Improve performance when filters in RowGroupFilter can match exactly
> --------------------------------------------------------------------
>
> Key: PARQUET-2237
> URL: https://issues.apache.org/jira/browse/PARQUET-2237
> Project: Parquet
> Issue Type: Improvement
> Reporter: Mars
> Priority: Major
>
> If we can accurately judge by the minMax status, we don’t need to load the
> dictionary from filesystem and compare one by one anymore.
> Similarly , Bloomfilter needs to load from filesystem, it may costs time and
> memory. If we can exactly determine the existence/nonexistence of the value
> from minMax or dictionary filters , then we can avoid using Bloomfilter to
> Improve performance.
> For example,
> # read data greater than {{x1}} in the block, if minMax in status is all
> greater than {{{}x1{}}}, then we don't need to read dictionary and compare
> one by one.
> # If we already have page dictionaries and have compared one by one, we
> don't need to read BloomFilter and compare.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
