zhaoyudi-creator opened a new issue, #19686: URL: https://github.com/apache/hudi/issues/19686
### Describe the problem you faced On table version < 8 (Timeline Layout V1, e.g. V5/V6/V7) MOR tables with Avro log blocks, written by Flink streaming with async instant generation, there is a deterministic data-loss race between compaction planning and an inflight deltacommit. For V5 MOR-Avro, the log file name token encodes the base instant of the owning file slice, while each log block header INSTANT_TIME carries the actual delta commit instant. The two are not the same. BaseHoodieCompactionPlanGenerator freezes the concrete list of log file paths enumerated at schedule time into the plan. Its file-level filter isCompletedBefore(D, logFile.getDeltaCommitTime()) uses the file name token (the base instant), so it cannot tell that some blocks inside a file belong to a still-inflight deltacommit. Because Flink async instant generation lets the coordinator hand out a new write instant before the previous one commits (write-commit / compaction-scheduling and next-instant-generation run on two independent executors, and MOR does not enable the instant-generation CommitGuard — OptionsResolver.isBlockingInstantGeneration returns true only for COW + UPSERT), a write instant C can be inflight while compaction D is scheduled (C < D). ### To Reproduce Consider one file group with this timeline: A — completed base instant (old slice) B — completed deltacommit, wrote *.A.log.*, belongs to the old slice C — inflight deltacommit; its file is also prefixed *.A.log.* (file name token = A, block header INSTANT_TIME = C), belongs to the old slice D — compaction plan scheduled while C is still inflight (C < D) Two sub-cases, both losing C's data after compaction finishes: Case 1 — C has not written its file yet when D is planned plan(D) enumerates the old slice as base A + log B only, without C's path. C later writes *.A.log.* and commits successfully, but the file is not in the plan. After compaction, the latest snapshot picks slice D and C's data is permanently invisible. Case 2 — C has written its file but is still inflight when D is planned Since the file name token = A is completed, the file-level filter includes *.A.log.* in the plan. But at compaction execution time C is still inflight, so block header INSTANT_TIME = C is unreadable and C's data is not merged into base D. C eventually commits, but its data is stranded in the old slice — same result as Case 1. ### Expected behavior Data written by deltacommit C must be visible in the latest snapshot after compaction D completes, regardless of the ordering between C's completion and D's scheduling. No committed deltacommit should be silently dropped from file-slice visibility. ### Environment Description * Hudi version: 1.1.1 * Spark version: * Flink version: 1.20 * Hive version: * Hadoop version: * Storage (HDFS/S3/GCS..): * Running on Docker? (yes/no): ### Additional context _No response_ ### Stacktrace ```shell ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
