sarvekshayr opened a new pull request, #9719: URL: https://github.com/apache/ozone/pull/9719
## What changes were proposed in this pull request? Ozone currently has no way to clear missing containers from the system. Even if all the data is deleted from the OM, the block deletes will never leave SCM because it has no replicas to send them to. As a short term mitigation, we added a CLI to SCM that supports “acking“ missing containers by ID if the admin confirms they are not a problem, so they do not mask future issues. This would remove them from `ozone admin container report` output and the missing container count metric. This would need to be persisted in the `ContainerInfo` in SCM, and we show this property in `ozone admin container info`. There is also a CLI to raise containers as an issue again and to query the list of acked missing containers. ## What is the link to the Apache JIRA [HDDS-14103](https://issues.apache.org/jira/browse/HDDS-14103) ## How was this patch tested? Container report shows 1 MISSING container. ``` bash-5.1$ ozone admin container report Container Summary Report generated at 2026-02-05T04:28:00Z ========================================================== Container State Summary ======================= OPEN: 0 CLOSING: 1 QUASI_CLOSED: 0 CLOSED: 0 DELETING: 0 DELETED: 0 RECOVERING: 0 Container Health Summary ======================== HEALTHY: 0 UNDER_REPLICATED: 0 MIS_REPLICATED: 0 OVER_REPLICATED: 0 MISSING: 1 UNHEALTHY: 0 EMPTY: 0 OPEN_UNHEALTHY: 0 QUASI_CLOSED_STUCK: 0 OPEN_WITHOUT_PIPELINE: 0 UNHEALTHY_UNDER_REPLICATED: 0 UNHEALTHY_OVER_REPLICATED: 0 MISSING_UNDER_REPLICATED: 0 QUASI_CLOSED_STUCK_UNDER_REPLICATED: 0 QUASI_CLOSED_STUCK_OVER_REPLICATED: 0 QUASI_CLOSED_STUCK_MISSING: 0 First 100 MISSING containers: #1 ``` Acknowledge the container as MISSING as it is not an issue. ``` bash-5.1$ ozone admin container ack 1 Acknowledged container: 1 bash-5.1$ ozone admin container ack --list 1 bash-5.1$ ozone admin container list [ { "state" : "CLOSING", "stateEnterTime" : "2026-02-05T04:22:50.850Z", "replicationConfig" : { "replicationFactor" : "ONE", "requiredNodes" : 1, "minimumNodes" : 1, "replicationType" : "RATIS" }, "usedBytes" : 12, "numberOfKeys" : 1, "lastUsed" : "2026-02-05T04:30:04.195255259Z", "owner" : "omServiceIdDefault", "containerID" : 1, "deleteTransactionId" : 0, "sequenceId" : 2, "healthState" : "HEALTHY", "open" : true, "deleted" : false, "ackMissing" : true } ] ``` Container report removes # 1 as MISSING. ``` bash-5.1$ ozone admin container report Container Summary Report generated at 2026-02-05T04:28:10Z ========================================================== Container State Summary ======================= OPEN: 0 CLOSING: 1 QUASI_CLOSED: 0 CLOSED: 0 DELETING: 0 DELETED: 0 RECOVERING: 0 Container Health Summary ======================== HEALTHY: 0 UNDER_REPLICATED: 0 MIS_REPLICATED: 0 OVER_REPLICATED: 0 MISSING: 0 UNHEALTHY: 0 EMPTY: 0 OPEN_UNHEALTHY: 0 QUASI_CLOSED_STUCK: 0 OPEN_WITHOUT_PIPELINE: 0 UNHEALTHY_UNDER_REPLICATED: 0 UNHEALTHY_OVER_REPLICATED: 0 MISSING_UNDER_REPLICATED: 0 QUASI_CLOSED_STUCK_UNDER_REPLICATED: 0 QUASI_CLOSED_STUCK_OVER_REPLICATED: 0 QUASI_CLOSED_STUCK_MISSING: 0 ``` Unacknowledge the container as MISSING as it is problematic. ``` bash-5.1$ ozone admin container unack 1 Unacknowledged container: 1 ``` Container report adds # 1 as MISSING again. ``` bash-5.1$ ozone admin container report Container Summary Report generated at 2026-02-05T04:28:30Z ========================================================== Container State Summary ======================= OPEN: 0 CLOSING: 1 QUASI_CLOSED: 0 CLOSED: 0 DELETING: 0 DELETED: 0 RECOVERING: 0 Container Health Summary ======================== HEALTHY: 0 UNDER_REPLICATED: 0 MIS_REPLICATED: 0 OVER_REPLICATED: 0 MISSING: 1 UNHEALTHY: 0 EMPTY: 0 OPEN_UNHEALTHY: 0 QUASI_CLOSED_STUCK: 0 OPEN_WITHOUT_PIPELINE: 0 UNHEALTHY_UNDER_REPLICATED: 0 UNHEALTHY_OVER_REPLICATED: 0 MISSING_UNDER_REPLICATED: 0 QUASI_CLOSED_STUCK_UNDER_REPLICATED: 0 QUASI_CLOSED_STUCK_OVER_REPLICATED: 0 QUASI_CLOSED_STUCK_MISSING: 0 First 100 MISSING containers: #1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
