furquan39 commented on issue #66301:
URL: https://github.com/apache/doris/issues/66301#issuecomment-5130415439
Answers to the requested items below.
## Corrections to our report
1. **`ADMIN SET REPLICA VERSION` behaves as designed.** We accept that the
command is FE-side bookkeeping only (journal + `Replica` object, no BE RPC),
and that the "revert" we reported is the expected `needSync` / tablet-report
reconciliation once the BE re-asserts its on-disk state. Our framing of it as a
broken repair mechanism was wrong. We agree the actionable part is
documentation (the command never repairs BE storage) plus an operator-facing
surface for missing version ranges.
2. **"Only repair is a full rebuild" was too strong.** We did not know
`skip_missing_version` and `pad_rowset` existed; that discoverability gap is
itself part of the problem we were trying to report, but the claim as written
was incorrect. We deliberately did **not** run `pad_rowset`: the damaged table
was renamed and parked immediately after the incident and remains in its broken
state for inspection, so all forensics below come from the untouched tablets.
## 1. DDL, key type, and the exact ADMIN SET REPLICA VERSION statements with
before/after state
Key type is `DUPLICATE KEY`, not unique/MoW:
`enable_unique_key_merge_on_write` is absent from the DDL, and the BE tablet
meta header (item 2) confirms `keys_type: "DUP_KEYS"` and
`"enable_unique_key_merge_on_write": false`. Replication is 1, which is why
clone recovery was impossible.
<details>
<summary>SHOW CREATE TABLE, verbatim (parked table)</summary>
```sql
CREATE TABLE `evaluated_docs_broken_20260728` (
`customer` varchar(128) NULL,
`product` varchar(128) NULL,
`market` varchar(32) NULL,
`created_at_short` date NULL,
`category` array<varchar(128)> NULL,
`doc_lang` varchar(16) NULL,
`created_at` datetime(3) NULL,
`last_updated_at` datetime(3) NULL,
`document` variant<PROPERTIES ("variant_max_subcolumns_count" =
"2048","variant_enable_typed_paths_to_sparse" =
"false","variant_max_sparse_column_statistics_size" =
"10000","variant_sparse_hash_shard_count" = "0")> NULL,
`evaluation_summary` variant<PROPERTIES ("variant_max_subcolumns_count" =
"2048","variant_enable_typed_paths_to_sparse" =
"false","variant_max_sparse_column_statistics_size" =
"10000","variant_sparse_hash_shard_count" = "0")> NULL,
`label_details` variant<PROPERTIES ("variant_max_subcolumns_count" =
"2048","variant_enable_typed_paths_to_sparse" =
"false","variant_max_sparse_column_statistics_size" =
"10000","variant_sparse_hash_shard_count" = "0")> NULL,
`kafka_topic` varchar(255) NULL,
`kafka_partition` int NULL,
`kafka_offset` bigint NULL,
`ingested_at` datetime(3) NULL DEFAULT CURRENT_TIMESTAMP,
`usage_status` array<varchar(64)> NULL,
`amounts_are_consistent` boolean NULL
) ENGINE=OLAP
DUPLICATE KEY(`customer`, `product`, `market`, `created_at_short`)
DISTRIBUTED BY HASH(`customer`) BUCKETS 8
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"min_load_replica_num" = "-1",
"is_being_synced" = "false",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V3",
"light_schema_change" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false",
"group_commit_interval_ms" = "10000",
"group_commit_data_bytes" = "134217728"
);
```
</details>
The exact statements we ran (one per affected tablet, all replicas on the
same backend):
```sql
ADMIN SET REPLICA VERSION PROPERTIES("tablet_id"="1781603714478",
"backend_id"="1784584274857", "last_failed_version"="-1");
ADMIN SET REPLICA VERSION PROPERTIES("tablet_id"="1781603714480",
"backend_id"="1784584274857", "last_failed_version"="-1");
ADMIN SET REPLICA VERSION PROPERTIES("tablet_id"="1781603714482",
"backend_id"="1784584274857", "last_failed_version"="-1");
ADMIN SET REPLICA VERSION PROPERTIES("tablet_id"="1781603714484",
"backend_id"="1784584274857", "last_failed_version"="-1");
ADMIN SET REPLICA VERSION PROPERTIES("tablet_id"="1781603714488",
"backend_id"="1784584274857", "last_failed_version"="-1");
ADMIN SET REPLICA VERSION PROPERTIES("tablet_id"="1781603714490",
"backend_id"="1784584274857", "last_failed_version"="-1");
ADMIN SET REPLICA VERSION PROPERTIES("tablet_id"="1781603714492",
"backend_id"="1784584274857", "last_failed_version"="-1");
```
Before/after: we set only `last_failed_version = -1` (we never touched
`version`). We do not have a captured replica-status snapshot from the seconds
immediately after the command; what we have is the durable end state, which is
the meaningful one: days later, `ADMIN SHOW REPLICA STATUS` again shows
`LastFailedVersion = 51947` on all 7 replicas. Since the BE can never publish
51947 over the hole, every publish retry fails and the failed-version flag
returns. That is fully consistent with your description of the report path, and
we withdraw the "revert" framing per the corrections above.
```
TabletId ReplicaId BackendId Version LastFailedVersion
LastSuccessVersion CommittedVersion VersionNum IsBad State Status
1781603714478 1785225810498 1784584274857 51946 51947
51946 51946 1 false NORMAL VERSION_ERROR
1781603714480 1785225469132 1784584274857 51946 51947
51946 51946 1 false NORMAL VERSION_ERROR
1781603714482 1785225810317 1784584274857 51946 51947
51946 51946 1 false NORMAL VERSION_ERROR
1781603714484 1785225810506 1784584274857 51946 51947
51946 51946 1 false NORMAL VERSION_ERROR
1781603714486 1785248005357 1784584274857 51946 -1
51946 51946 4 false NORMAL OK
1781603714488 1785225469239 1784584274857 51946 51947
51946 51946 1 false NORMAL VERSION_ERROR
1781603714490 1785225807157 1784584274857 51946 51947
51946 51946 1 false NORMAL VERSION_ERROR
1781603714492 1785225469236 1784584274857 51946 51947
51946 51946 1 false NORMAL VERSION_ERROR
```
Note `IsBad = false` and `State = NORMAL` throughout; `VERSION_ERROR` in
this ADMIN command is the only operator-visible signal, supporting the
surfacing improvement you propose.
## 2. Tablet forensics: meta header and compaction show
7 of 8 tablets are affected (all except 1781603714486, the one tablet that
holds the skewed data; the affected ones are zero-row tablets that still
advance through every published version). On every affected tablet: FE says
`Version = CommittedVersion = 51946` with `LastFailedVersion = 51947`, while
the BE meta header's rowset chain ends at 51945. The hole is exactly
`[51946-51946]`.
`/api/meta/header/1781603714478` (key fields; schema block elided):
```json
{
"table_id": 1781603714476,
"partition_id": 1781603714475,
"tablet_id": 1781603714478,
"schema_hash": 541545333,
"tablet_state": "PB_RUNNING",
"enable_unique_key_merge_on_write": false,
"rs_metas": [
{ "start_version": 0, "end_version": 51945, "num_rows": 0,
"rowset_state": "VISIBLE", "creation_time": 1785369639 }
]
}
```
`/api/compaction/show?tablet_id=1781603714478`:
```json
{
"cumulative policy type": "size_based",
"cumulative point": 51926,
"last cumulative status": "[E-2000]candidate_rowsets is empty",
"last base status": "[OK]",
"last base success time": "2026-07-30 00:00:39.726",
"rowsets": [
"[0-51945] 0 DATA NONOVERLAPPING
020000000002611d2d43f8bd0b7e149bbc7dc78c5ec3368b 0"
],
"missing_rowsets": [],
"stale_rowsets": [],
"stale version path": []
}
```
Note `missing_rowsets` is empty: the hole at the top of the chain is
invisible in the compaction view and only detectable by comparing the max
`end_version` against the FE version, or via the BE log lines in item 3. That
is more support for an explicit operator-facing missing-range surface.
**Important caveat:** a routine base compaction ran successfully on these
tablets at 2026-07-30 00:00:39 UTC (matching the surviving rowset's
`creation_time`) and merged the pre-incident chain, which was internally
contiguous below the hole, into the single `[0-51945]` rowset. So the original
multi-rowset chain shape at crash time is no longer inspectable on disk. What
is preserved, and parked, is the boundary itself: nothing on the BE covers
version 51946 while the FE insists 51946 is committed. A healthy sibling tablet
shows the contrast:
```
tablet 1781603714486 rs_metas (healthy):
[0-1] rows=0 VISIBLE
[2-25503] rows=84890 VISIBLE
[25504-44501] rows=43608 VISIBLE
[44502-51946] rows=20895 VISIBLE <- contiguous through 51946,
matching FE
```
The PVC is preserved and `pad_rowset` has not been run.
## 3. BE logs
Logs from before the post-incident restart are lost: the BE pod was
recreated at 23:13 UTC on incident day (2026-07-28), and the log directory
contains only files created at or after 2026-07-28 23:13. The crash moment and
the original publish/commit of version 51946 are therefore not in any retained
log (listed as a gap below).
Within the surviving logs, grepping the affected tablet IDs for `publish
txn`, `add committed rowset`, `add visible rowset` returns nothing: after
restart the BE never again received or committed a rowset for these tablets.
Starting about two minutes after restart, and repeating roughly once per minute
per publish retry of 51947, the hole signature appears:
```
W20260728 23:15:12.247043 1526 rowset_version_mgr.cpp:80] missed version
for version_range=[0-51946], tablet_id=1781603714478, st=[INTERNAL_ERROR]fail
to find path in version_graph. spec_version: 0-51946
W20260728 23:15:12.247124 1526 base_tablet.cpp:318] 1781603714478 has 1
missed version:[51946-51946],
W20260728 23:16:13.838908 1553 rowset_version_mgr.cpp:80] missed version
for version_range=[0-51946], tablet_id=1781603714478, st=[INTERNAL_ERROR]fail
to find path in version_graph. spec_version: 0-51946
W20260728 23:16:13.838980 1553 base_tablet.cpp:318] 1781603714478 has 1
missed version:[51946-51946],
```
The same signature also hit tablets of two other RF1 tables and one small
state table hosted on the same BE (28 replicas total across 4 tables, all on
the dead node's BE).
## 4. FE logs
Not extracted in this pass. The FE-side facts quoted here come from live FE
metadata (which persisted the incident independently: `LstFailedTime =
2026-07-28 15:00:05`, FE local time, on every affected replica marks when
publishing to the dead BE first failed). We will check FE log retention for the
incident window (publish task completion/resend, tablet reports, the admin
commands) and attach the excerpts if retained; otherwise this becomes part of
the controlled reproduction below.
## 5. Trigger evidence: node loss, not container OOM
This incident is the node/storage-durability branch of your analysis, not
the process-only kill branch. Our original report's framing around a kubelet
SIGKILL/OOM should be read accordingly; the evidence points at host-level node
death:
- The worker node hosting the BE shows all kubelet conditions with
`lastTransitionTime = 2026-07-28T14:09:51Z` and `Ready` transitioning at
`2026-07-28T14:10:21Z`: standard residue of a NotReady episode with
re-registration on the incident day. A second (control-plane) node shows a host
restart the following morning and a lost role label, consistent with the same
disruption event.
- The BE pod object was recreated (creationTimestamp
`2026-07-28T23:13:06Z`), so it has `restartCount = 0` and `lastState = {}`.
There is no `lastState.terminated` OOMKilled record, and none would be
expected: the host died under the pod; the kernel did not kill the container.
- Kubernetes events for the NotReady/reboot window have aged out (default 1
h TTL); node condition residue is what remains.
- `sync_tablet_meta`: confirmed `false` (the default) on all three BEs via
`/api/show_config`:
```json
[["sync_tablet_meta","bool","false","true"]]
```
That combination (host power-off class failure + un-fsynced tablet-meta
RocksDB writes) is consistent with your plausibility ranking: an acknowledged
recent meta write evaporated at the block/page-cache layer while the FE's own
fsynced edit log kept the publish.
Storage layer: BE storage is Ceph RBD on a 3x replicated pool via the Ceph
CSI driver. The PVC survived the node death and was re-attached by the
recreated pod, which is why everything through version 51945 is intact; only
the in-flight meta write was lost. The damage is purely at the Doris
tablet-meta level, not the block layer.
## 6. Exact missing range and absence of any committed rowset artifact
- Missing range: exactly `[51946-51946]` on every affected tablet.
- The meta header JSON for the affected tablets contains only `rs_metas`,
all `VISIBLE`. There is no pending or committed rowset array of any kind in the
returned document, and the compaction endpoint shows `"stale_rowsets": []`. The
transaction the FE published as version 51946 left **no rowset artifact of any
state** on the BE, and no segment files for it exist on disk. So the
FE-resend/republish recovery route you describe had nothing to work with: the
committed rowset meta itself did not survive.
## Gaps and follow-ups
What only a controlled reproduction can produce, and what we commit to:
1. BE logs spanning the crash moment (pre-restart logs did not survive the
pod recreation) and the FE log excerpts if retention fails: a hardened re-run
environment is being prepared, and we will run the fault-injection shape there.
2. Your reproduction plan matches our incident shape exactly: a fault point
for one tablet after `make_visible` of version N+1 and before
`RowsetMetaManager::save` of version N, run for both non-MoW and MoW tables,
with `sync_tablet_meta` both false and true, and with separate process-only
kill and simulated node/storage durability loss variants. The node-loss +
`sync_tablet_meta=false` variant is the one our forensics correspond to.
3. The parked table remains available in its broken state; we are happy to
run further read-only queries or endpoint dumps against it on request.
4. We are equally happy to validate a candidate fix (publish durability
ordering or an operator-facing missing-range surface) on our cluster and report
back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
