ArafatKhan2198 opened a new pull request, #12096: URL: https://github.com/apache/ozone/pull/12096
## What changes were proposed in this pull request? When Recon brings a deleted container back to match SCM, it deletes its own record and adds a new one. Deleting the record also deletes the list of datanodes that hold copies. Recon then treats the container as missing even though the data is still there. SCM makes the same move in place and keeps that list. This change makes Recon do the same. **What goes wrong** 1. Recon reads a page of containers that SCM has marked deleted. 2. Before Recon finishes that page, a datanode reports a copy that still has data. SCM brings the container back to QUASI_CLOSED or CLOSED. 3. Recon finishes the old page and marks the container deleted anyway. It still knows which datanodes hold copies. 4. On the next sync, SCM lists the container as alive. `rebuildContainerFromScm` deletes Recon's record and adds a new one from SCM's container info. 5. The new record has no replicas. The next health check, every 5 minutes by default, marks the container MISSING. It stays MISSING until each datanode's next full report, up to 60 minutes by default, adds the replicas back. No data is lost. 6. If adding the new record fails, the failure is only logged and the container is gone from Recon. An old MISSING row can still be in the unhealthy-containers table when no health check ran between the two syncs, for example after a restart. `/api/v1/containers/unhealthy` then returns HTTP 500 for every page that includes that row, until a later sync adds the container back. At the default interval that is 6 hours. **Change** `rebuildContainerFromScm` still checks that SCM lists the container in that state. It then calls `transitionDeletingOrDeletedToTargetState`, the same in-place move SCM's report handler uses, instead of deleting and re-adding the record. The replicas stay. If the update fails, the container stays in Recon as DELETED with its replicas, the unhealthy-containers API keeps working, and the next sync tries again. Recon keeps its own container details (sequence id, key count, used bytes). Replica reports already keep those up to date. Recon can still mark the container deleted from the stale page. It shows DELETED until the next sync moves it back. Skipping a missing container in the unhealthy-containers API, instead of failing the page, is left as separate hardening. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17061 ## How was this patch tested? Ran the `TestReconSCMContainerSyncIntegration` in `:ozone-recon` and had a green run - ```bash mvn -pl :ozone-recon test -Dtest=TestReconSCMContainerSyncIntegration -DskipShade -DskipRecon ``` - `containerDeletedInReconButClosedInScmKeepsReplicas`: Recon has the container DELETED with one replica, and SCM lists it CLOSED. After sync the container is CLOSED and the replica is still there. - `failedCorrectionFromDeletedKeepsContainerInRecon`: the in-place update throws. The container stays in Recon as DELETED, with its replica. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
