LemonCL commented on issue #68120:
URL: https://github.com/apache/doris/issues/68120#issuecomment-5711085686
Incident evidence from a production incident (downstream build based on
branch-3.1;
the meta-service code in this area is identical to master). All timestamps
are from
the same incident, IPs and cluster names are masked.
**Scenario**: scale-in of a compute group, 28 nodes decommissioned in
parallel.
**1. MS log — every node is deleted then re-added in two separate
transactions
(~4ms apart, note the status flip DECOMMISSIONING → DECOMMISSIONED):**
```
I20260916 15:06:13.303140 resource_manager.cpp:1118] node to modify
json={"cloud_unique_id":"1:<instance>:VpS2Z3xR","ip":"10.x.x.103",...,"status":"NODE_STATUS_DECOMMISSIONING",...}
I20260916 15:06:13.307343 resource_manager.cpp:1118] node to modify
json={"cloud_unique_id":"1:<instance>:VpS2Z3xR","ip":"10.x.x.103",...,"status":"NODE_STATUS_DECOMMISSIONED",...}
```
**2. FE log — its 10s poll landed exactly in that window for one node,
dropped
the backend, and re-registered it with a new backend id on the next poll:**
```
2026-09-16 15:06:13,311 INFO (cloud tablet rebalancer)
[CloudTabletRebalancer.checkDecommissionState():607] prepare to notify meta
service be 10.x.x.239:9050 decommissioned
2026-09-16 15:06:13,358 INFO (cloud cluster check)
[CloudSystemInfoService.updateCloudBackendsUnLock():305] dropped cloud
backend=Backend [id=1788251570096, host=10.x.x.239, ...]
2026-09-16 15:06:23,403 INFO (cloud cluster check)
[CloudSystemInfoService.updateCloudBackendsUnLock():281] added cloud
backend=Backend [id=1788252118413, host=10.x.x.239, ...]
```
Same host, same BE process (identical process epoch), but backend id changed
1788251570096 → 1788252118413.
**3. The subsequent DROP_NODE for a node in this group then failed with:**
```
I20260916 15:06:43.083244 meta_service_helper.h:226] finish alter_cluster
... response=status { code: CLUSTER_NOT_FOUND msg: "not found ,can not find to
drop nodes by cloud_unique_id=... help Msg=Ms nodes memory cache may be
inconsistent..." }
```
**4. Timeline (correlated):**
| time | FE | MS |
|---|---|---|
| 15:06:13.311 | rebalancer sends NOTIFY_DECOMMISSIONED for node .239 | |
| 15:06:13.30x–.31x | | txn1: node deleted from nodes[]; txn2: re-added as
DECOMMISSIONED |
| 15:06:13.358 | checker poll sees node missing → drops backend id ...570096
| |
| 15:06:23.403 | checker re-registers node as new backend id ...118413 | |
| 15:06:43.083 | | DROP_NODE fails: CLUSTER_NOT_FOUND |
Afterwards the rebalancer treated the re-registered node as a fresh idle BE
and
moved ~120k tablets back onto it, and the decommission workflow could not
complete (the compute group stayed in "changing" state).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
