ArafatKhan2198 opened a new pull request, #12095:
URL: https://github.com/apache/ozone/pull/12095

   ## What changes were proposed in this pull request?
   
   A quasi-closed container has stopped taking writes, but its copies do not 
yet agree on the last data. SCM keeps it in QUASI_CLOSED until that is settled. 
This change makes Recon follow SCM when SCM moves such a container back to 
QUASI_CLOSED and Recon is still showing it as CLOSED.
   
   **What goes wrong**
   
   1. The container is CLOSED and empty. SCM and Recon both show CLOSED. One 
datanode holding an older, non-empty copy is offline.
   2. SCM deletes the empty copies and moves the container to DELETING, then 
DELETED. Recon does not copy DELETING, and it copies DELETED only on its 
container sync, which defaults to every 6 hours. Recon still shows CLOSED.
   3. The datanode comes back and reports its copy, for example as UNHEALTHY. 
SCM sees data in a container it was deleting and moves the container back to 
QUASI_CLOSED.
   4. Recon receives the same report, but its report handler does nothing for a 
container it already has as CLOSED.
   5. On the next sync, SCM no longer lists the container as DELETED, so that 
pass misses it. The QUASI_CLOSED pass finds it, then skips it, because Recon 
has it CLOSED. Every later sync skips it the same way.
   6. Recon keeps showing the container as CLOSED and leaves it out of the 
QUASI_CLOSED list and count for as long as SCM keeps it QUASI_CLOSED. SCM and 
the datanodes are not affected.
   
   The skip is there for a real case: Recon can be ahead of SCM, because it 
closed the container from a CLOSED replica that SCM has not processed yet. 
Moving that container back would be wrong. SCM never goes from CLOSED straight 
to QUASI_CLOSED. The only path is through DELETING or DELETED. So when SCM says 
QUASI_CLOSED and Recon says CLOSED, Recon is ahead only if it still has a 
CLOSED replica recorded. Otherwise SCM is ahead, and that is this bug.
   
   **Change**
   
   In `ReconStorageContainerSyncHelper.reconcileToQuasiClosed`, a container 
that SCM lists as QUASI_CLOSED and Recon has as CLOSED is moved to QUASI_CLOSED 
when none of its replicas in Recon is CLOSED. The move uses the same two steps 
SCM uses, DELETE and then 
`transitionDeletingOrDeletedToTargetState(QUASI_CLOSED)`, so the container 
record and its replicas are kept. A container with a CLOSED replica is still 
left CLOSED.
   
   If Recon stops between those two steps, the container is left DELETING. The 
next sync finishes that move, because SCM is listing the container as 
QUASI_CLOSED. The correction still happens on the next container sync.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-17060
   
   ## How was this patch tested?
   
   `TestReconSCMContainerSyncIntegration`: 45 tests, 0 failures.
   
   ```bash
   mvn -pl :ozone-recon test -Dtest=TestReconSCMContainerSyncIntegration 
-DskipShade -DskipRecon
   ```
   
   - `followsScmBackToQuasiClosedWhenReconHasNoClosedReplica`: Recon has the 
container CLOSED with only an UNHEALTHY replica. After sync it is QUASI_CLOSED 
and the replica is kept.
   - `doesNotOverwriteContainerAlreadyClosedInRecon`: Recon has a CLOSED 
replica. After sync the container stays CLOSED.
   - `finishesQuasiClosedMoveStoppedInDeleting`: Recon is left in DELETING. The 
next sync moves it to QUASI_CLOSED.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to