ArafatKhan2198 opened a new pull request, #12095: URL: https://github.com/apache/ozone/pull/12095
## What changes were proposed in this pull request? A quasi-closed container has stopped taking writes, but its copies do not yet agree on the last data. SCM keeps it in QUASI_CLOSED until that is settled. This change makes Recon follow SCM when SCM moves such a container back to QUASI_CLOSED and Recon is still showing it as CLOSED. **What goes wrong** 1. The container is CLOSED and empty. SCM and Recon both show CLOSED. One datanode holding an older, non-empty copy is offline. 2. SCM deletes the empty copies and moves the container to DELETING, then DELETED. Recon does not copy DELETING, and it copies DELETED only on its container sync, which defaults to every 6 hours. Recon still shows CLOSED. 3. The datanode comes back and reports its copy, for example as UNHEALTHY. SCM sees data in a container it was deleting and moves the container back to QUASI_CLOSED. 4. Recon receives the same report, but its report handler does nothing for a container it already has as CLOSED. 5. On the next sync, SCM no longer lists the container as DELETED, so that pass misses it. The QUASI_CLOSED pass finds it, then skips it, because Recon has it CLOSED. Every later sync skips it the same way. 6. Recon keeps showing the container as CLOSED and leaves it out of the QUASI_CLOSED list and count for as long as SCM keeps it QUASI_CLOSED. SCM and the datanodes are not affected. The skip is there for a real case: Recon can be ahead of SCM, because it closed the container from a CLOSED replica that SCM has not processed yet. Moving that container back would be wrong. SCM never goes from CLOSED straight to QUASI_CLOSED. The only path is through DELETING or DELETED. So when SCM says QUASI_CLOSED and Recon says CLOSED, Recon is ahead only if it still has a CLOSED replica recorded. Otherwise SCM is ahead, and that is this bug. **Change** In `ReconStorageContainerSyncHelper.reconcileToQuasiClosed`, a container that SCM lists as QUASI_CLOSED and Recon has as CLOSED is moved to QUASI_CLOSED when none of its replicas in Recon is CLOSED. The move uses the same two steps SCM uses, DELETE and then `transitionDeletingOrDeletedToTargetState(QUASI_CLOSED)`, so the container record and its replicas are kept. A container with a CLOSED replica is still left CLOSED. If Recon stops between those two steps, the container is left DELETING. The next sync finishes that move, because SCM is listing the container as QUASI_CLOSED. The correction still happens on the next container sync. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17060 ## How was this patch tested? `TestReconSCMContainerSyncIntegration`: 45 tests, 0 failures. ```bash mvn -pl :ozone-recon test -Dtest=TestReconSCMContainerSyncIntegration -DskipShade -DskipRecon ``` - `followsScmBackToQuasiClosedWhenReconHasNoClosedReplica`: Recon has the container CLOSED with only an UNHEALTHY replica. After sync it is QUASI_CLOSED and the replica is kept. - `doesNotOverwriteContainerAlreadyClosedInRecon`: Recon has a CLOSED replica. After sync the container stays CLOSED. - `finishesQuasiClosedMoveStoppedInDeleting`: Recon is left in DELETING. The next sync moves it to QUASI_CLOSED. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
