ravjot28 opened a new pull request, #20617:
URL: https://github.com/apache/hudi/pull/20617
### Describe the issue this Pull Request addresses
Closes #20611
With `hoodie.bloom.index.use.metadata=true` and the metadata table's bloom
filter partition built,
`HoodieKeyLookupHandle` asks `HoodieTableMetadata#getBloomFilter(partition,
fileName)` for the file id instead of the
base file name the partition is keyed by. Every lookup misses and the write
fails with "BloomFilter missing". The
handle serves `ListBasedHoodieBloomIndexHelper`, the bloom index helper of
the Java write client and of the Flink
write client's `BLOOM` and `GLOBAL_BLOOM` indexes. Spark's metadata path
(`HoodieMetadataBloomFilterProbingFunction`)
already asks by base file name and is not changed.
### Summary and Changelog
Bloom index lookups through `ListBasedHoodieBloomIndexHelper` read bloom
filters from the metadata table.
- `HoodieKeyLookupHandle` resolves the latest base file of its file group
once, in the constructor, and uses it for
the bloom filter (from the metadata table by `HoodieBaseFile#getFileName`,
or from the file footer as before) and
for the key check in `getLookupResult`, which resolved it a second time
before. The filter and the keys now come
from the same base file.
- New `TestJavaBloomIndexWithMetadataBloomFilters` (Java client,
copy-on-write and merge-on-read): with the metadata
table's bloom filter partition built and
`hoodie.bloom.index.use.metadata=true`, every update is tagged to its file
group after the first write, after a copy-on-write rewrite of the base
files or a merge-on-read compaction (new
base file names, same file ids), and after clustering (new file groups; no
update goes to a replaced one).
Related: draft #20263 also keeps the base file in a field of
`HoodieKeyLookupHandle` (to reuse the one the driver
resolved) and still passes the file id to the metadata table; #13450 changes
which base file the handle resolves
while a compaction is pending. Both touch the same lines; whichever lands
second rebases, and the file name lookup
applies to the base file either chooses.
### Impact
No API or config change. Java bloom index lookups with
`hoodie.bloom.index.use.metadata=true` (and Flink's, which use the
same handle) read the
bloom filters from the metadata table instead of failing. With the option
off, the handle reads the same base file
as before and asks the file system view for it once instead of twice.
**Backport:** please include this fix in the next 1.2.x patch release. The
commit cherry-picks cleanly onto
`release-1.2.1`.
### Risk Level
low. The metadata lookup now uses the key the metadata table is written
with; the footer path reads the same base
file as before.
Tests, run on master with JDK 11 and passing checkstyle checks for main and
test sources, with
`mvn test -pl hudi-client/hudi-java-client,hudi-client/hudi-flink-client -am
-Dtest=...`:
- Without the main code change: both cases of
`TestJavaBloomIndexWithMetadataBloomFilters` (copy-on-write and
merge-on-read) fail on the first upsert with `HoodieIndexException:
BloomFilter missing for <file id>`.
- With the change: `TestJavaBloomIndexWithMetadataBloomFilters` 2/2,
`TestFlinkHoodieBloomIndex` 25/25,
`TestJavaHoodieBackedMetadata` 68/68,
`TestHoodieJavaClientOnCopyOnWriteStorage` 39/39 pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]