hudi-agent commented on code in PR #19590: URL: https://github.com/apache/hudi/pull/19590#discussion_r3764295331
########## website/docs/performance.md: ########## @@ -132,6 +132,48 @@ To enable Data Skipping in your queries make sure to set following properties to - `hoodie.metadata.enable` (to enable metadata table use on the read path, enabled by default) - `hoodie.metadata.index.column.stats.enable` (to enable column stats index use on the read path) +#### Parquet Bloom Filters + +Column stats prune on ranges, so they help least where they are needed most: an equality predicate on a +high-cardinality column whose min-max range covers almost every file. Parquet's own bloom filters cover that +case. They are written into the Parquet file itself, and a reader consults them to skip row groups that +cannot contain the value being searched for. + +Hudi passes these through to the Parquet writer, per column, from the **Hadoop** configuration: + +| key | meaning | +| --- | --- | +| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for `<column>` | +| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct values, which sizes the filter | + +Set them on the Hadoop configuration your writer uses. From Spark, the `spark.hadoop.` prefix forwards them: Review Comment: 🤖 In the example, `--conf spark.hadoop.parquet.bloom.filter.enabled#uuid=true` uses `uuid` as the column, which reads as though it's a Hudi record-key column. Since the preceding paragraph is about high-cardinality equality predicates on arbitrary data columns, it might help to use a neutral column name (or add one sentence clarifying that `<column>` is any data column you filter on by equality), so readers don't conflate this with the record-key bloom index you carefully distinguish in the note just below. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## website/docs/performance.md: ########## @@ -132,6 +132,48 @@ To enable Data Skipping in your queries make sure to set following properties to - `hoodie.metadata.enable` (to enable metadata table use on the read path, enabled by default) - `hoodie.metadata.index.column.stats.enable` (to enable column stats index use on the read path) +#### Parquet Bloom Filters + +Column stats prune on ranges, so they help least where they are needed most: an equality predicate on a +high-cardinality column whose min-max range covers almost every file. Parquet's own bloom filters cover that Review Comment: 🤖 This section is placed under **Read Path** and states that "a reader consults them to skip row groups," but everything documented below is write-side (the `parquet.bloom.filter.enabled#<column>` / `expected.ndv` writer keys). It might help to spell out what the *reader* needs for these filters to take effect — Parquet bloom-filter row-group filtering has to be enabled on the read side (e.g. `parquet.filter.bloom.enabled=true`) and depends on predicate pushdown, and Spark's vectorized reader hasn't always honored bloom filters across versions. As written, a user could set the two write keys, see no query speedup, and reasonably conclude the docs are wrong. @yihua could you confirm whether Hudi's Spark read path actually consults Parquet bloom filters out of the box, and what (if any) reader-side config is required? That would let us either document the read step or reframe the section as write-side only. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
