yujun777 opened a new pull request, #68390: URL: https://github.com/apache/doris/pull/68390
### What problem does this PR solve? Issue Number: N/A Related PR: #68170, #68180, #68193 Trace issue: https://github.com/apache/doris/issues/65418 Problem Summary: An IVM MV keeps rows that a metadata-only base-table change (`DROP` / `TRUNCATE` / `REPLACE` / `RECOVER PARTITION`) has made unusable, because such a change emits no row binlog and nothing incremental can remove those rows. Today the invalidation is recorded at MV granularity: `IvmInfo.completeBaselineRebuildRequired` / `pendingBaselineRebuildPartitions` plus a `schemaChangeVersion` guard. That granularity is coarse -- one dirty partition drags the whole MV to a COMPLETE refresh, a task result produced before the invalidation is discarded, and a strict `REFRESH ... INCREMENTAL` is rejected until a COMPLETE refresh has run, even when the change touched nothing the MV reads. This PR replaces the barrier with a per-MV-partition requirement: * `MTMV.partitionStates` maps each MV partition to `{refreshEpoch, latestEpoch}` (persisted as `pst`, journaled through `ALTER_PARTITION_STATES`). A partition is dirty iff `latestEpoch > refreshEpoch && refreshEpoch != 0` -- `refreshEpoch == 0` means it was never refreshed, so it holds no rows and its first refresh reads the current base tables anyway. * An invalidation that can be placed on the MV partitions reading the changed base partitions raises only their requirement and drops their refresh snapshot, in one journal record. Every other partition keeps catching up incrementally, and a task result that no invalidation reached is no longer discarded. * Alignment runs after partition sync and before any base table is read, so "the partition exists" and "the entry exists" are the same thing and a mark always has an entry to land on. * A refresh samples the requirement per batch before reading base tables, and writes back only `refreshEpoch` once that batch's data is committed, so an invalidation arriving mid-refresh is not swallowed. * The refresh routes on the criterion: dirty partitions are rebuilt by the partition executor (under a strict INCREMENTAL request as well), the rest are caught up incrementally or skipped. When every partition needs a rebuild, or the MV is in `SCHEMA_CHANGE`, the refresh runs as COMPLETE. A refresh that rebuilt partitions the request did not ask for reports how many in the new `IvmRebuiltPartitions` column of the mv task TVF. * A whole-MV invalidation -- the fallback that cannot place a change on any partition, and a property change that widens what the MV maintains -- now goes through the MV state instead of the barrier flag, and the barrier fields, the refresh-time guard and the pending-rebuild rejection are gone. A rename of an IVM MV's base table no longer moves it into that state. Behaviour changed: Yes * A strict `REFRESH ... INCREMENTAL` that meets an invalidated baseline no longer fails with "IVM baseline rebuild is pending"; it rebuilds (the whole MV for a schema-level invalidation, the invalidated partitions otherwise) and reports the count in `IvmRebuiltPartitions`. * An MV in `SCHEMA_CHANGE` refreshes as COMPLETE without first attempting the incremental rewrite, so `IvmFallbackReason` stays unset where it previously reported the barrier's label. * A base-table rename no longer moves an IVM MV out of `NORMAL`. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test - FE unit tests: `IvmBaselineRebuildTest`, `MTMVTaskTest`, `MTMVTest`, `CreateMTMVCommandTest`, `MTMVRelationManagerTest`, `AlterMTMVTest`, `IvmInfoTest`, `MTMVRefreshSnapshotTest`, `MTMVPartitionUtilTest`, `MetaLockUtilsTest` -- 280 tests, all pass. Every new case was checked to fail when the change it covers is reverted. - Regression: `mtmv_p0/ivm` -- `test_ivm_partition_epoch_rebuild` (new), `test_ivm_baseline_marker_scope`, `test_ivm_partition_baseline_rebuild`, `test_ivm_partition_baseline_rebuild_dup_keys`, `test_ivm_partition_sync_limit`, `test_ivm_chained_mtmv_1`, `test_ivm_chained_mtmv_2` and `test_ivm_strict_incremental_rebuilds_invalidated_partitions` (renamed from `test_ivm_strict_failure_partition_atomicity`), all pass with the stored expectations compared, not regenerated. - Behavior changed: Yes (see above) - Does this need documentation: No (the user-facing notes and the design doc update are part of a follow-up) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
