ravjot28 opened a new issue, #20611:
URL: https://github.com/apache/hudi/issues/20611

   ### Bug Description
   
   **What happened:**
   
   With the metadata table's bloom filter index built 
(`hoodie.metadata.index.bloom.filter.enable=true`) and
   `hoodie.bloom.index.use.metadata=true`, every upsert through 
`HoodieJavaWriteClient` with `hoodie.index.type=BLOOM`
   fails once the table has a base file: the key lookup asks the metadata table 
for a bloom filter that is never found,
   and the write fails with `HoodieIndexException: BloomFilter missing for 
<file id>`. The same lookup serves the Flink
   write client's `BLOOM` and `GLOBAL_BLOOM` indexes 
(`FlinkHoodieIndexFactory`). Spark is not affected.
   
   **What you expected:**
   
   The bloom filters are read from the metadata table, and the upsert tags its 
records as it does with
   `hoodie.bloom.index.use.metadata=false`.
   
   **Steps to reproduce:**
   
   1. Create a copy-on-write or merge-on-read table with 
`HoodieJavaWriteClient`, `hoodie.index.type=BLOOM`,
      `hoodie.metadata.enable=true`, 
`hoodie.metadata.index.bloom.filter.enable=true` and
      `hoodie.bloom.index.use.metadata=true`.
   2. `insert` some records and commit; the metadata table now has the 
`bloom_filters` partition.
   3. `upsert` new versions of some of those records.
   4. The upsert fails with `HoodieIndexException: BloomFilter missing for 
<file id>`.
   
   **Root cause (master a65a573a49f):**
   
   - The `bloom_filters` partition of the metadata table is keyed by partition 
and base file name: the writer creates
     each record with the base file name 
(`hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:335-341`,
     `createBloomFilterMetadataRecord(partitionName, baseFileName, ...)`), and
     `HoodieTableMetadata#getBloomFilter(partitionName, fileName)` reads it by 
file name
     
(`hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadata.java:175-184`).
   - `HoodieKeyLookupHandle#getBloomFilter`
     
(`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieKeyLookupHandle.java:65-66`)
 passes
     `partitionPathFileIDPair.getRight()`, the file id, so the lookup never 
matches and `orElseThrow` raises
     "BloomFilter missing".
   - The handle is created only by `HoodieBloomIndexCheckFunction`
     
(`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/HoodieBloomIndexCheckFunction.java:98,107`),
     which `ListBasedHoodieBloomIndexHelper` uses; `JavaHoodieIndexFactory` and 
`FlinkHoodieIndexFactory` build their
     bloom indexes on that helper. Spark's metadata path, 
`HoodieMetadataBloomFilterProbingFunction`
     
(`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/bloom/HoodieMetadataBloomFilterProbingFunction.java:105,125`),
     resolves the latest base file of each file group and asks for its file 
name, so it works.
   - The file id has been passed here since the metadata bloom filter index was 
added (0.11.0).
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at a65a573a49f), 1.2.1 and earlier
   **Query engine:** Java client (`HoodieJavaWriteClient`); Flink write client 
with a `BLOOM` or `GLOBAL_BLOOM` index (same code path, not run)
   **Relevant configs:** `hoodie.index.type=BLOOM`, 
`hoodie.metadata.enable=true`,
   `hoodie.metadata.index.bloom.filter.enable=true`, 
`hoodie.bloom.index.use.metadata=true`
   
   ### Logs and Stack Trace
   
   ```
   org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit 
time <instant>
        at 
org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:83)
        at 
org.apache.hudi.table.action.commit.JavaUpsertCommitActionExecutor.execute(JavaUpsertCommitActionExecutor.java:46)
        at 
org.apache.hudi.client.HoodieJavaWriteClient.upsert(HoodieJavaWriteClient.java:128)
        ...
   Caused by: org.apache.hudi.exception.HoodieIndexException: BloomFilter 
missing for <file id>
        at 
org.apache.hudi.io.HoodieKeyLookupHandle.getBloomFilter(HoodieKeyLookupHandle.java:66)
        at 
org.apache.hudi.io.HoodieKeyLookupHandle.<init>(HoodieKeyLookupHandle.java:55)
        at 
org.apache.hudi.index.bloom.HoodieBloomIndexCheckFunction$LazyKeyCheckIterator.next(HoodieBloomIndexCheckFunction.java:98)
        at 
org.apache.hudi.index.bloom.ListBasedHoodieBloomIndexHelper.findMatchingFilesForRecordKeys(ListBasedHoodieBloomIndexHelper.java:71)
        at 
org.apache.hudi.index.bloom.HoodieBloomIndex.lookupIndex(HoodieBloomIndex.java:130)
        at 
org.apache.hudi.index.bloom.HoodieBloomIndex.tagLocation(HoodieBloomIndex.java:87)
        at 
org.apache.hudi.table.action.commit.JavaWriteHelper.tag(JavaWriteHelper.java:57)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to