rangareddy opened a new pull request, #19590:
URL: https://github.com/apache/hudi/pull/19590
### Describe the issue this Pull Request addresses
Closes #16063 (HUDI-6456), "Add parquet blooms documentation".
Hudi forwards Parquet's per-column bloom filter settings through to the
Parquet writer, but the only trace of
bloom filters in the docs was one auto-generated row in `configurations.md`
— and that row is for a **different
feature** with a near-identical name. So nothing told a user that these keys
exist, that they are per column,
or that they are read from the Hadoop configuration:
```
parquet.bloom.filter.enabled#<column>
parquet.bloom.filter.expected.ndv#<column>
```
### Summary and Changelog
Adds a **Parquet Bloom Filters** subsection to `performance.md`, under `Read
Path` next to `Data Skipping`,
because that is precisely the gap it fills: column stats prune on ranges and
so help least for an equality
predicate on a high-cardinality column whose min-max spans nearly every file
— which is what a bloom filter
covers.
The section states the two keys, that they are per column, how to set them
from Spark (`spark.hadoop.` prefix),
how to size `expected.ndv`, and that it is a write-time decision so an
existing table only picks them up as it
is rewritten.
Two caveats are called out, because both are easy to get wrong and neither
was written down anywhere:
1. **`hoodie.parquet.bloom.filter.enabled` is not this feature.** It
controls whether Hudi writes a bloom
filter **of record keys** into the footer for the [bloom
index](https://hudi.apache.org/docs/indexes) during
upserts. It defaults to `true`, applies only when meta fields are
populated, and is implied anyway when
`hoodie.index.type` names a `BLOOM` index. Setting it does **not** enable
per-column Parquet filters. Given
the names differ by one prefix, a user reading `configurations.md` could
reasonably conclude the opposite.
2. **The settings are applied reflectively and failure is swallowed.** On a
Parquet without
`withBloomFilterEnabled` / `withBloomFilterNDV`, the keys are silently
ignored rather than rejected.
### Verification
Docs change, so no test to add — said plainly rather than implied. Every
claim was read off master
(`3ba31dd37fff`):
| claim | source |
| --- | --- |
| keys are read from the **Hadoop** config, per column |
`HoodieBaseParquetWriter.handleParquetBloomFilters`, `:98-119` — iterates the
`Configuration` and splits the column off each key |
| exact key spelling | `HoodieBaseParquetWriter:52-53`, and
`TestHoodieParquetBloom.scala:36-37` sets
`parquet.bloom.filter.enabled#bloom_col` / `expected.ndv#bloom_col` on
`jsc.hadoopConfiguration` |
| forwarded via `withBloomFilterEnabled` / `withBloomFilterNDV` | `:105` and
`:114`, invoked by reflection |
| silently ignored on an older Parquet | `:107-109` and `:116-118` —
`NoSuchMethodException` is caught and skipped |
| `hoodie.parquet.bloom.filter.enabled` is about record-key blooms for the
bloom index | `HoodieStorageConfig:252-258` (default `true`, since 0.15.0) and
`HoodieFileWriterFactory.enableBloomFilter:141-146` (`populateMetaFields &&
(thisConfig ‖ index.type contains BLOOM)`) |
| Flink sets them the same way |
`TestHoodieRowDataParquetConfigInjector:100-101` |
Markdown checked by running `markdownlint` over both files **before and
after** and comparing rule *classes*
rather than counts, since these files already carry findings:
```
before: MD007 MD009 MD012 MD013 MD031 MD033 MD040
after : MD007 MD009 MD012 MD013 MD031 MD033 MD040 -> no new class of
finding
```
Also checked: every `<column>` occurrence sits inside inline code, so
nothing new is exposed to the MDX
parser; the `indexes.md` link target exists in **both** doc trees so the
relative link resolves in the
versioned copy too; and the inserted region is byte-identical between the
two files.
**Not done:** the Docusaurus build (`website/node_modules` absent, full
install heavy). The section adds a
table, a fenced block and two admonitions, all constructs already used on
this page.
### Impact
Documentation only — no code, config, API or format change. Two features
whose names differ by one prefix are
now distinguishable, and a write-side knob that previously required reading
`HoodieBaseParquetWriter` to
discover is now documented.
### Risk Level
none
### Documentation Update
This is the documentation update. Targets `asf-site`; applied to `docs/` and
`versioned_docs/version-1.2.0/`, matching the file set of the recently
merged #19572. Older versioned copies
are frozen snapshots and are left alone.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable — n/a for docs; verification
table above instead
- [ ] CI passes on my PR — `asf-site` PRs do not run the `master` gates;
`markdownlint` parity checked locally
as described
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
