Doris-Breakwater commented on issue #68807: URL: https://github.com/apache/doris/issues/68807#issuecomment-6073721217
Breakwater-GitHub-Analysis-Slot: slot_c5c3cef4e05c ### Initial assessment This warrants storage/compaction correctness triage with high priority because the reported single-replica impact removes the only eligible load target. **There is a source-supported gap in the level-2 selection invariant, but the cause of the affected production tablets is not yet established.** The issue currently has no labels; storage/compaction and correctness labels would be appropriate if available. I inspected upstream release **4.0.8** at `bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36` and **4.1.4** at `ad35a140c7fd0b842f18c23300bac581f7d04326`. Neither is asserted to match the running binary. No Doris code was modified and no Doris binary reproduction was run. I could not verify an existing fix for this precise mechanism; this does not establish that none exists. ### Verified source behavior 1. `CumulativeCompaction::pick_rowsets_to_compact()` first restricts candidates to a consecutive version path, **then** calls the policy's `pick_input_rowsets()`. It does not revalidate the nonempty filtered selection before returning success. Thus candidate continuity does not imply final-input continuity. See [4.0.8 selection](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/cumulative_compaction.cpp#L179-L300). 2. The level-2 loop stops at a level-0 rowset, but **continues past** a level-1 rowset whose age is at most 24 hours. It can collect later eligible rowsets on the other side of that skipped range. Publication still depends on reaching a trigger: at least two selected rowsets and either the level-2 size target (10 times the effective level-1 goal) or the time target (10 times the table time threshold). Earlier ordinary-compaction branches can also return before this loop. The same filtering pattern exists in both inspected releases. See [4.0.8 policy](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/cumulative_compaction_time_series_policy.cpp#L268-L436) and [4.1.4 policy](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/compaction/cumulative_compaction_time_series_policy.cpp#L267-L435). 3. `build_basic_info()` constructs the output interval from the first selected start and last selected end. `merge_input_rowsets()` supplies readers for the selected rowsets. Its row-count check compares the output, merged and filtered rows against **selected-input** rows; it cannot detect an omitted rowset solely from that count. See [output interval](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/compaction.cpp#L466-L512) and [correctness check](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/compaction.cpp#L1504-L1520). 4. The local replacement path holds the tablet header lock and verifies that selected inputs still exist with matching rowset IDs. Those checks protect against changed inputs, but do not establish adjacency or exclude an unselected active rowset inside the output interval. `Tablet::modify_rowsets()` removes the selected versions and inserts the output; `TabletMeta::modify_rs_metas()` likewise leaves unselected metadata intact. See [commit path](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/compaction.cpp#L1440-L1467), [replacement](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/tablet.cpp#L530-L630), and [metadata replacement](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/tablet_meta.cpp#L1003-L1031). I also checked a **policy-only simulation** of the supplied ranges with nonempty, nonsingleton level-1 rowsets, ages old/young/old, a 1 MiB goal and 6 MiB per rowset. The ordinary pass has no starting singleton/empty rowset; the level-2 size gate selects the two outer ranges and derives `[2-120]`. This assumes an existing cumulative point at or below 2, visible local candidates and no delete predicates. It demonstrates the filtering/range mismatch under explicit guards; it does **not** show how a real tablet acquires that age ordering or that a Doris merge commits it. ### Bad-replica and load path There is a concrete source path consistent with the reported symptoms: the BE examines active metadata for version crossing; for a running tablet, crossing causes a report with `used=false`. FE report handling can then mark the replica bad. Load target selection excludes bad replicas even on live backends, and `OlapTableSink` rejects an insufficient eligible-replica count. See [BE overlap detection](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/tablet.cpp#L1141-L1165), [BE report](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/tablet.cpp#L1605-L1683), [FE propagation](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java#L1335-L1350), [replica filtering](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java#L252-L277), a nd [load count check](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java#L770-L801). Logs are still needed to establish that this path caused the actual flag transition; IO/path failures can also produce bad reports. ### What remains unproven / requested evidence - Exact FE/BE versions and build Git SHAs, deployment mode, sanitized table definition/key model, effective compaction properties and relevant BE settings. Match source to the actual build before choosing a fix or upgrade. - For one affected tablet, before/after active and stale rowset inventories, with consistent anonymized IDs, actual version boundaries, compaction levels, creation times relative to one common reference, segment/row counts, sizes, delete predicates, cumulative point and visible version. Preserve real relationships rather than substituting unrelated synthetic values. - BE compaction records identifying the **actual selected input IDs** and output ID/range, trigger, start/success times, and any ordered-compaction path. Correlate these with the first overlap report, FE bad-replica transition, tablet health details and exact Stream Load error. - Minimal binary reproduction, including load/empty-transaction sequence, compaction timing and any clone, schema-change, restart or clock-adjustment activity. Writers stamp creation time when producing rowsets; version order alone does not establish age order ([writer](https://github.com/apache/doris/blob/bc8ea1bac6d62d92cdebd2e3ac33cc668961ec36/be/src/olap/rowset/beta_rowset_writer.cpp#L125-L140)). Empty outputs receive level 0 in this policy, which stops the level-2 scan; empty rowsets alone do not prove the proposed hole. Cumulative-compaction locking and input-ID validation also mean concurrency must be demonstrated, not assumed. ### Suggested next steps and recovery Maintainers should reproduce the old/young/old selection using real rowsets and production triggers, then test rejection of a gap after filtering and immediately before metadata replacement. An output must exactly cover a consecutive selected chain and must not intersect any unselected active rowset. Include tests for young-prefix skipping, internal skips, empty/level-0 boundaries and changed inputs. Merely changing one `continue` needs an explicit selection policy and regression coverage. Setting `time_series_compaction_level_threshold=1` bypasses the inspected level-2 branch, so it is a plausible containment measure **after confirming the running implementation and effective tablet metadata**. It has not been validated here, does not cancel an already prepared compaction, and does not repair existing overlaps. For affected single-replica tablets, first coordinate preservation of tablet metadata and all surviving active/stale rowset files before further mutation or stale-rowset cleanup. Retain the original evidence. Clearing `isBad`, dropping a replica, deleting metadata or forcing compaction does not establish a complete readable version chain. Normal replica cloning requires a valid source; the sole bad replica provides no verified healthy donor. Prefer restoring a known-good backup into a separate table or reloading retained source data, then validate coverage and contents. If neither exists, maintainers must assess surviving original input files and metadata on a copy and provide a build-specific recovery procedure. The public evidence is insufficient to prescribe safe in-place metadata surgery or promise lossless recovery. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
