yujun777 opened a new pull request, #67814: URL: https://github.com/apache/doris/pull/67814
### What problem does this PR solve? `partition_sync_limit` keeps only a recent slice of each base table's partitions, so an MV can be missing a partition its base table still has. The incremental delta read that base partition anyway, through the delta scan and through the join-opposite snapshot. A change to a non-partitioned dimension therefore produced delta rows for dates the MV has no partition for, and the insert failed with `no partition for this tuple`. Re-syncing cannot bring an expired partition back, so the retry loop in `executeIvmAttempt` could never recover: the task failed after exhausting its attempts, and because the write is atomic, the partitions the MV does keep were left unrepaired as well. A strict `REFRESH ... INCREMENTAL`, and a scheduled refresh of an MV declared without `FALLBACK`, fail outright. ### What changed The incremental delta now reads only the base partitions the MV's partition definition keeps. - `MTMVPartitionUtil.generateRelatedBasePartitionIds` returns that partition set per base table partitioned by the MV's partition column, and an empty value when `partition_sync_limit` is not set, which is the only property that can leave the MV without a base partition. - `IvmIncrRefreshManager` passes it to `IvmRewriteContext.incremental`, and `IvmDeltaRewriter` applies it as an upper bound on the partitions each base table may be read from, intersected with `ivm_partition_window_limit` when that is set too. - The bound is the partition set the MV is aligned to rather than the partitions it already has, so a base partition whose MV partition partition sync has yet to add stays readable: the refresh still reports the missing partition and recovers it by syncing, which is what keeps a newly added base partition working. - A base table absent from the set keeps its full read, since the MV partition column does not come from it and limiting it would change join results without narrowing the target MV partitions. A set covering every partition of its table leaves the plan untouched, so an MV that mirrors all of its base partitions is unaffected. - Only olap tables are reachable: the delta reads them through their stream and restricts a scan by partition id, which a connector table has no equivalent of. The COMPLETE paths are unchanged. The PR also removes the `nonConcurrent` group from seven `mtmv_p0/ivm` suites that use neither debug points nor global variables or config, so they run in the parallel pool again. ### Test - `regression-test/suites/mtmv_p0/ivm/test_ivm_partition_sync_limit.groovy` (new): the MV keeps only a recent slice of the base partitions, and a late-arriving dimension row must repair the partition it keeps without failing on the one it dropped. Partitions are added by hand with literal dates and no dynamic-partition scheduler, and the expectation does not depend on the run date. - `regression-test/suites/mtmv_p0/ivm/test_ivm_partition_sync_limit_with_window.groovy` (new): `partition_sync_limit` and `ivm_partition_window_limit` together, the readable base partitions are their intersection. - `testIncrementalScopeRestrictsDeltaToScopePartitions`, `testIncrementalScopeCoveringAllPartitionsLeavesScanUnchanged`, `testIncrementalScopeWithoutPartitionsProducesEmptyRelation` - `testHasPartitionSyncLimit`, `testGenerateRelatedBasePartitionIdsWithoutSyncLimit`, `testGenerateRelatedBasePartitionIdsOnSelfManageMv`, `testGenerateRelatedBasePartitionIdsWithoutMvPartitionInfo` - `test_ivm_partition_sync_limit`, `test_ivm_partition_sync_limit_with_window`, `MTMVPartitionUtilTest`, `IvmDeltaRewriterTest`, `IvmIncrRefreshManagerTest` Trace issue: https://github.com/apache/doris/issues/65418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
