ravjot28 opened a new pull request, #20617:
URL: https://github.com/apache/hudi/pull/20617

   ### Describe the issue this Pull Request addresses
   
   Closes #20611
   
   With `hoodie.bloom.index.use.metadata=true` and the metadata table's bloom 
filter partition built,
   `HoodieKeyLookupHandle` asks `HoodieTableMetadata#getBloomFilter(partition, 
fileName)` for the file id instead of the
   base file name the partition is keyed by. Every lookup misses and the write 
fails with "BloomFilter missing". The
   handle serves `ListBasedHoodieBloomIndexHelper`, the bloom index helper of 
the Java write client and of the Flink
   write client's `BLOOM` and `GLOBAL_BLOOM` indexes. Spark's metadata path 
(`HoodieMetadataBloomFilterProbingFunction`)
   already asks by base file name and is not changed.
   
   ### Summary and Changelog
   
   Bloom index lookups through `ListBasedHoodieBloomIndexHelper` read bloom 
filters from the metadata table.
   
   - `HoodieKeyLookupHandle` resolves the latest base file of its file group 
once, in the constructor, and uses it for
     the bloom filter (from the metadata table by `HoodieBaseFile#getFileName`, 
or from the file footer as before) and
     for the key check in `getLookupResult`, which resolved it a second time 
before. The filter and the keys now come
     from the same base file.
   - New `TestJavaBloomIndexWithMetadataBloomFilters` (Java client, 
copy-on-write and merge-on-read): with the metadata
     table's bloom filter partition built and 
`hoodie.bloom.index.use.metadata=true`, every update is tagged to its file
     group after the first write, after a copy-on-write rewrite of the base 
files or a merge-on-read compaction (new
     base file names, same file ids), and after clustering (new file groups; no 
update goes to a replaced one).
   
   Related: draft #20263 also keeps the base file in a field of 
`HoodieKeyLookupHandle` (to reuse the one the driver
   resolved) and still passes the file id to the metadata table; #13450 changes 
which base file the handle resolves
   while a compaction is pending. Both touch the same lines; whichever lands 
second rebases, and the file name lookup
   applies to the base file either chooses.
   
   ### Impact
   
   No API or config change. Java bloom index lookups with 
`hoodie.bloom.index.use.metadata=true` (and Flink's, which use the
   same handle) read the
   bloom filters from the metadata table instead of failing. With the option 
off, the handle reads the same base file
   as before and asks the file system view for it once instead of twice.
   
   **Backport:** please include this fix in the next 1.2.x patch release. The 
commit cherry-picks cleanly onto
   `release-1.2.1`.
   
   ### Risk Level
   
   low. The metadata lookup now uses the key the metadata table is written 
with; the footer path reads the same base
   file as before.
   
   Tests, run on master with JDK 11 and passing checkstyle checks for main and 
test sources, with
   `mvn test -pl hudi-client/hudi-java-client,hudi-client/hudi-flink-client -am 
-Dtest=...`:
   - Without the main code change: both cases of 
`TestJavaBloomIndexWithMetadataBloomFilters` (copy-on-write and
     merge-on-read) fail on the first upsert with `HoodieIndexException: 
BloomFilter missing for <file id>`.
   - With the change: `TestJavaBloomIndexWithMetadataBloomFilters` 2/2, 
`TestFlinkHoodieBloomIndex` 25/25,
     `TestJavaHoodieBackedMetadata` 68/68, 
`TestHoodieJavaClientOnCopyOnWriteStorage` 39/39 pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to