rangareddy commented on code in PR #19590: URL: https://github.com/apache/hudi/pull/19590#discussion_r3764335190
########## website/docs/performance.md: ########## @@ -132,6 +132,48 @@ To enable Data Skipping in your queries make sure to set following properties to - `hoodie.metadata.enable` (to enable metadata table use on the read path, enabled by default) - `hoodie.metadata.index.column.stats.enable` (to enable column stats index use on the read path) +#### Parquet Bloom Filters + +Column stats prune on ranges, so they help least where they are needed most: an equality predicate on a +high-cardinality column whose min-max range covers almost every file. Parquet's own bloom filters cover that +case. They are written into the Parquet file itself, and a reader consults them to skip row groups that +cannot contain the value being searched for. + +Hudi passes these through to the Parquet writer, per column, from the **Hadoop** configuration: + +| key | meaning | +| --- | --- | +| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for `<column>` | +| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct values, which sizes the filter | + +Set them on the Hadoop configuration your writer uses. From Spark, the `spark.hadoop.` prefix forwards them: Review Comment: Applied in `41a3a0d`. You are right that `uuid` was a poor choice directly above a note distinguishing this from the record-key bloom index — it invited exactly the conflation the note exists to prevent. The example now uses `session_id`, and I took the second half of your suggestion too rather than relying on the reader to infer it: > `<column>` is any data column you filter on by equality — not the record key, and unrelated to the record-key bloom index discussed in the note below. Both changes are in all six copies of the page. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
