furquan39 opened a new issue, #66301: URL: https://github.com/apache/doris/issues/66301
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Apache Doris 4.1.0-rc03. This report cites no file:line anchors; we did not trace the publish path in source. ### Environment We operate Doris 4.1.0-rc03 on Kubernetes with Ceph-backed storage. BE pods can be hard-killed by the kubelet (OOM, node failure), which is the trigger here. ### What's Wrong? A hard BE process death (SIGKILL by the kubelet, no shutdown hook) that lands in the middle of version publish can leave the on-disk tablet meta with a hole in the rowset version chain. After the BE restarts, any read or write that needs a continuous version path on the affected tablet fails with: ``` fail to find path in version_graph. spec_version: 0-51946 ``` The observed sequence: 1. BE is publishing versions on a tablet under normal ingest load. 2. The kubelet kills the BE pod mid-publish (hard death, no graceful shutdown). 3. The BE restarts and reloads tablet meta from disk. The persisted rowset set has a gap: the version graph cannot build a continuous path from version 0 to the visible version. 4. All reads and writes against the tablet are blocked with the error above. Compaction on the tablet also fails permanently. The natural operator response is to reset the replica's reported version on the FE side: ``` ADMIN SET REPLICA VERSION ... ``` This does not stick. The FE-side metadata change is reverted as soon as the BE sends its next tablet report, because the BE re-reports its actual on-disk state and the on-disk state still has the hole. There appears to be no supported path, FE-side or BE-side, that reconciles the persisted rowset meta itself. With replication factor 1 there is no healthy peer to clone from, so the tablet cannot self-heal. The only repair we found is a full rebuild of the table (drop and reload from source). The same terminal state is reachable at RF2 or RF3 via a double fault: if the last healthy replica dies while a peer is already broken, no clone source remains. ### What You Expected? Either of: 1. The tablet meta update during publish is crash-consistent, so a hard death at any instruction boundary leaves the on-disk rowset chain in a state from which the version graph can always be rebuilt (old chain or new chain, never a torn intermediate). 2. If a hole is possible, the BE has a repair path that reconstructs a usable version graph from the surviving rowsets on disk, accepting bounded data loss for the missing versions, instead of leaving the tablet permanently unreadable and unwritable. We also expected `ADMIN SET REPLICA VERSION` to either take effect durably or to fail with a clear message explaining that the BE on-disk state will override it. ### How to Reproduce? The window is narrow, so this is timing-dependent, but the sequence that produced it for us: 1. Create a table with `replication_num = 1` and drive sustained ingest so that version publish is continuously active on its tablets. 2. Hard-kill the BE process mid-ingest (SIGKILL, or in Kubernetes let the kubelet kill the pod, for example via an OOM or forced node drain). Do not allow a graceful shutdown. 3. Restart the BE. 4. Query or write to the table. The affected tablet fails with `fail to find path in version_graph. spec_version: 0-<visible_version>` (in our case `0-51946`). 5. Attempt `ADMIN SET REPLICA VERSION` to flag-repair the replica. Observe that the FE state reverts after the next BE tablet report. Repeated hard kills under load raise the hit probability. We have not found a way to make step 2 deterministic from the outside; a fault-injection point inside the publish path would. ### Anything Else? Impact in production terms: the affected tablets blocked both reads and writes until we rebuilt the table from the upstream source. With `replication_num = 1` this turns a single unclean pod death into data-loss-equivalent downtime for the tablet, and at RF2/RF3 it converts a double fault into the same terminal state. Suggested directions, in case they are useful: 1. Make the rowset chain transition during publish atomic with respect to hard crashes (write new meta, fsync, then switch, so recovery always finds a complete chain). 2. Add a BE-side salvage mode that rebuilds the version graph from whatever rowsets survive on disk, marking the tablet's visible version down to the highest continuous version, so the tablet becomes usable again with explicit, bounded loss. 3. Improve operator tooling for this state: make `ADMIN SET REPLICA VERSION` either durable against BE re-reports or explicitly rejected with a message pointing at the on-disk inconsistency, and surface "version graph has holes" as a distinct tablet health state in `SHOW TABLETS` / `ADMIN SHOW REPLICA STATUS` so it can be detected before a query trips over it. Prior reports: #36832, #42021, and #49524 share the same `fail to find path in version_graph` error signature, spanning versions from 2.0.x to 4.1.x, and all were closed stale without a root cause or fix. None identifies the crash-mid-publish trigger or the `ADMIN SET REPLICA VERSION` revert behavior described here. We are happy to provide full BE logs from the incident and to test patches. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's Code of Conduct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
