yihua commented on code in PR #19802:
URL: https://github.com/apache/hudi/pull/19802#discussion_r4213531275
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -523,16 +523,14 @@ String generateUniqueInstantTime(String
initializationTime) {
protected abstract EngineType getEngineType();
- private Lazy<List<FileSliceAndPartition>> getLazyMergedFileSlices() {
+ private Lazy<List<FileSliceAndPartition>> getLazyMergedFileSlices(String
maxInstantTime) {
Review Comment:
After further investigation, I found that the implementation of async
indexer has deviated from the design over time and there are gaps in the
current implementation that can cause consistency issue (thus changes in the
this PR which touches this part, though not solving the problem fully).
Particularly, on master, regular writers use the table-config inflight
marker (`hoodie.table.metadata.partitions.inflight`) to detemine which metadata
partitions to update, which `buildMetadataPartitions` from async indexer writes
at run time outside the lock, instead of checking the pending indexing request
at schedule time, and the indexer catch-up skips any instant that already has
an MDT deltacommit instead of verifying that it wrote to the new partition. A
commit between schedule and execute is missed by column stats and bloom
filters, and a writer whose commit overlaps the start of the run is never
indexed (RLI loses all of that commit's keys; under LAZY the catch-up waits for
it and then applies nothing). For example,
```
t0 indexer: constructs its metadata writer, takes timeline snapshot S (X
is INFLIGHT in S)
t1 writer A: builds its metadata writer at commit time, reads table config
-> record_index neither completed nor inflight ->
partitionsToUpdate = {files}
t2 indexer: writes the inflight marker (no lock)
t3 writer A: writes MDT deltacommit X with files only, completes X,
releases the lock
t4 indexer: bootstraps from snapshot S -> X excluded (pending in S)
t5 catch-up: X has a metadata deltacommit -> skipped (EAGER: "Ignoring
instant ... no heartbeat";
LAZY: waits for X to complete, then applies nothing)
```
HUDI-6200 also moved file-group creation into the indexer run (no longer
under a lock), so a writer committing during the run's listing phase now fails
on "FileGroup count for MDT partition ... should be > 0" or sees partial set of
file groups in index partition, leading to corrupted metadata.
That is also likely why the change of `getLazyMergedFileSlices(String
maxInstantTime)` to use indexing instant time to fence off the file system view
to restore original async indexer design, but this itself does not solve the
problem fully due to the two issues above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
