LemonCL opened a new issue, #68120: URL: https://github.com/apache/doris/issues/68120
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master, branch-3.1 (cloud mode) ### What's Wrong? In cloud mode meta-service, `handle_decommission_node` and `handle_notify_decommissioned` (cloud/src/meta-service/meta_service_resource.cpp) flip a node's status by calling `ResourceManager::modify_nodes` twice in two separate FDB transactions: first deleting the node from `InstanceInfoPB.nodes[]`, then re-adding it with the new status (DECOMMISSIONING / DECOMMISSIONED). Between the two committed transactions, the node is genuinely absent from the cluster PB. Any concurrent reader of `get_cluster`/`get_instance` (e.g. FE CloudClusterChecker, which polls every 10s) that lands in this window observes a phantom node removal. Consequences observed in production during a 28-node scale-in: - FE dropped the backend from memory, then re-registered it as a brand-new backend (new backend id) on the next poll; - The tablet rebalancer treated it as a fresh idle BE and moved ~120k tablets back onto it; - The subsequent DROP_NODE failed, leaving the decommission workflow wedged (compute group stuck in "changing" state). ### What You Expected? Node status transitions should be atomic: a reader sees either the old status or the new status, never "node missing". ### How to Reproduce? Inject a delay (e.g. sleep a few seconds) between the two `modify_nodes` calls in `handle_notify_decommissioned`, then concurrently call `get_cluster` — the node disappears from the response during the window. ### Anything Else? Proposed fix: replace the two `modify_nodes` calls (delete + re-add) with a single `ResourceManager::update_cluster` call that mutates the node's status in place within one FDB transaction — the same pattern already used by `handle_rename_cluster` / `handle_set_cluster_status`. PR: #68119 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
