yujun777 opened a new pull request, #67802: URL: https://github.com/apache/doris/pull/67802
### What problem does this PR solve? Dropping a base-table partition marks the IVM baseline as broken, because those rows disappear through metadata rather than through row binlog entries. Partition sync then removes the very MV partitions that barrier named, so `handlePendingIvmBaselineRebuild` intersected the barrier with the current MV partitions and always got an empty set. The refresh reported SUCCESS with refresh mode `NOT_REFRESH` and cleared the barrier without refreshing anything, leaving the delta that had accumulated on the surviving partitions unapplied: the task claimed the MV was up to date while the MV was missing rows. Reproduction: drop a base partition, insert a row into a surviving partition, then run `REFRESH MATERIALIZED VIEW ... INCREMENTAL FALLBACK`. The task succeeds, the expired MV partition is dropped, and the new row never reaches the MV. ### What changed - `MTMVTask.handlePendingIvmBaselineRebuild` is a pre-step now instead of a terminal branch. It rebuilds only the barrier partitions that still exist, releases the barrier, and then lets the normal attempt list run, so the surviving partitions catch up in the same task. Partitions the barrier named that partition sync already dropped need no rebuild: the partition and its IVM offsets are both gone, which is what satisfies those entries. - The attempt list is rewritten in place. A lone `COMPLETE` attempt rebuilds everything anyway and needs no pre-rebuild; a complete baseline rebuild rewrites the list to `COMPLETE` instead of executing it inline. - `MTMV.releaseIvmBaselineRebuild` releases the barrier under a `schemaChangeVersion` compare-and-clear and journals the new state immediately, like the other `ivmInfo` mutations. Without the release the IVM attempt that follows would be rejected by `validateIvmRefreshStart`; without the compare-and-clear a concurrent base-table change would lose the barrier entry it had just recorded. - `MTMVRelationManager.markIvmBaselineRebuild` takes an explicit all-partitions-changed flag instead of inferring it from an empty partition map. The COMPLETE paths are unchanged. They end the task, so the existing `addTaskResult` cleanup releases the barrier once the refresh succeeds. A strict `REFRESH ... INCREMENTAL`, and a scheduled refresh of an MV declared without `FALLBACK`, still fail while the barrier is pending, because that check runs before partition sync. Relaxing it is left to a follow-up PR. ### Test - `regression-test/suites/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.groovy` (new): partitions added and dropped by hand with literal dates and no dynamic-partition scheduler, so the case is deterministic. Asserts the MV matches the base table right after `INCREMENTAL FALLBACK`. The scenario was reproduced on the pre-fix build, where the fallback task reported SUCCESS while the MV was missing the row written to the surviving partition. - `regression-test/suites/mtmv_p0/ivm/test_ivm_partition_baseline_rebuild_dup_keys.groovy` (new): covers the `TRUNCATE` path, where the affected MV partition survives and the pre-step really has something to rebuild. Runs on a duplicate-key MV with repeated identical rows, so a delta applied twice would show up as extra rows instead of hiding behind a unique key. - `test_ivm_partition_baseline_rebuild`, `test_ivm_partition_sync_retry`, `test_ivm_partition_unique_key`, `test_ivm_partition_window_limit`, `test_ivm_partition_window_remove`, `test_ivm_one_row_relation_partitioned`, `test_ivm_strict_failure_partition_atomicity` Trace issue: https://github.com/apache/doris/issues/65418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
