[ https://issues.apache.org/jira/browse/PARQUET-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706489#comment-17706489 ]
ASF GitHub Bot commented on PARQUET-2254: ----------------------------------------- yabola commented on PR #1042: URL: https://github.com/apache/parquet-mr/pull/1042#issuecomment-1488873270 Calculate the size of nvd according to the size of bytes, I also saw similar logic in impala https://github.com/apache/impala/blob/2c779939dc302be9ee5dd97ddf374bb043040891/be/src/kudu/util/block_bloom_filter.cc#L258-L288 > Build a BloomFilter with a more precise size > -------------------------------------------- > > Key: PARQUET-2254 > URL: https://issues.apache.org/jira/browse/PARQUET-2254 > Project: Parquet > Issue Type: Improvement > Reporter: Mars > Assignee: Mars > Priority: Major > > h3. Why are the changes needed? > Now the usage of bloom filter is to specify the NDV(number of distinct > values), and then build BloomFilter. In general scenarios, it is actually not > sure how much the distinct value is. > If BloomFilter can be automatically generated according to the data, the file > size can be reduced and the reading efficiency can also be improved. > h3. What changes were proposed in this pull request? > {{DynamicBlockBloomFilter}} contains multiple {{BlockSplitBloomFilter}} as > candidates and inserts values in the candidates at the same time. Use the > largest bloom filter as an approximate deduplication counter, and then remove > incapable bloom filter candidates during data insertion. -- This message was sent by Atlassian Jira (v8.20.10#820010)