yujun777 opened a new pull request, #68180: URL: https://github.com/apache/doris/pull/68180
### What problem does this PR solve? Trace issue: https://github.com/apache/doris/issues/65418 Problem Summary: An IVM materialized view has to fully rebuild every MV partition that may hold rows of a base-table partition that was dropped / truncated / replaced / recovered. Those operations change the table through metadata and emit no row binlog, so the incremental path can never remove the stale rows: if such an MV partition is not marked, the orphan rows stay in the MV forever, with no error anywhere. Which MV partitions to mark was decided by asking the refresh snapshot which of them had *seen* the changed base partition (`MTMVRefreshSnapshot#getMvPartitionNames`). A snapshot is captured before the base table is read, so it is only a lower bound: a base partition that appears in the base table after the capture is not in it, and each partition's snapshot is written back by whichever task finishes last. When at least one other MV partition did match, the lookup returned that subset and the partition that had actually read the dropped one was silently left out: ``` partition p1: capture snapshot p1 reads the base table p1 still running base table: X is added p2 refreshes (its snapshot has X) X dropped marker: p1's snapshot has no X, p2's has -> only p2 is marked result: p1 keeps the rows of X forever ``` This asks the MV partition mapping instead: which MV partitions read the changed base partition is exactly the question, it is metadata rather than a lagging record, and it is the same mapping the refresh already uses for partition sync. | case | before | after | | --- | --- | --- | | MV partitions that read the changed base partition | marked | marked (unchanged) | | changed base partition read by no MV partition | whole MV rebuilt | nothing marked | | SELF_MANAGE MV, base table is not a PCT table, `RECOVER PARTITION`, a PCT table of a multi-PCT MV, mapping computation fails | whole MV rebuilt | whole MV rebuilt (unchanged) | The second row is intentional: the old lookup could not tell "no MV partition reads this partition" apart from "the snapshot does not know it", so it rebuilt the whole MV -- a table whose dynamic partitions are dropped ahead of the MV's partition sync forced a full rebuild every time. The mapping tells them apart, and the cases that genuinely cannot be answered from it are explicit checks rather than a guess from an empty result: - `RECOVER PARTITION` marks before the partition is added back to the table, so the partition is still in the recycle bin at that moment and no metadata-derived mapping can describe it. - The mapping is seeded from the MV's PCT tables and never gains a table later, so a joined partition table the MV's partition column does not reach is not described by it at all. - A mapping computation failure must not fail the base table DDL: it warns and rebuilds the whole MV. - The mapping reads the partition items of the MV and of every PCT table, so it takes their read locks, while the caller already holds the changed table's write lock: two concurrent partition DDLs on two PCT tables of the same MV would each hold the write lock the other one needs. Those reads are therefore taken with a bounded `tryReadLock` (`Table.TRY_LOCK_TIMEOUT_MS`, the same way the stream cleanup treats a busy table) and never block: while the other tables are free -- the normal case -- the selection still narrows to the MV partitions that read the changed base partition, and while one of them is being written the whole MV is rebuilt. Acquiring in id order cannot help here, because the first lock of the pair is already held before this point. The mapping is computed *before* the MV lock is taken, not inside it: computing it takes the partition items of the MV and of its PCT tables, and the MV lock is a leaf lock -- nothing is acquired under it today, and the lock analysis of the baseline barrier depends on that. The selection does not need to be atomic with the barrier it produces: the barrier is recorded under the lock, and the names it carries are intersected with the live partition names when they are consumed (`MTMVTask`). `MTMVRefreshSnapshot#getMvPartitionNames`, the old selection, is removed with the unit test that covered it; it has no other caller. ### Release note None ### Check List (For Author) - Test - [x] Regression test <!-- new suite: mtmv_p0/ivm/test_ivm_baseline_marker_scope --> - [x] Unit Test <!-- IvmBaselineRebuildTest: mapping selection, the four fallbacks, no-op case --> - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [ ] No. - [x] Yes. <!-- A changed base partition that no MV partition reads no longer forces a full rebuild --> - Does this need documentation? - [ ] No. - [x] Yes <!-- Same as before: the user-facing behavior of base-table partition changes is unchanged --> 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
