devmadhuu opened a new pull request, #10074:
URL: https://github.com/apache/ozone/pull/10074
## What changes were proposed in this pull request?
This PR addresses the issue of large deviations between Recon and SCM
related to container counts, container Ids and their respective states.
Largely the PR addresses two issues:
1. Recon may completely miss containers that SCM knows about.
2. Recon may know the container, but keep it in an older lifecycle state
such as:
`OPEN`, `CLOSING`, or `QUASI_CLOSED` after SCM has already advanced it.
The implementation now has two SCM sync mechanisms:
1. Full snapshot sync
This remains the safety net. Recon replaces its SCM DB view from a fresh
SCM
checkpoint on the existing snapshot schedule.
2. Incremental targeted sync
This now runs on its own schedule and decides between:
- `NO_ACTION`
- `TARGETED_SYNC`
- `FULL_SNAPSHOT`
The targeted sync is implemented as a four-pass workflow:
1. Pass 1: `CLOSED`
Add missing `CLOSED` containers and correct stale
`OPEN`/`CLOSING`/`QUASI_CLOSED`
containers to `CLOSED`.
2. Pass 2: `OPEN`
Add missing `OPEN` containers only. No downgrades and no state correction.
3. Pass 3: `QUASI_CLOSED`
Add missing `QUASI_CLOSED` containers and correct stale `OPEN`/`CLOSING`
containers up to `QUASI_CLOSED`.
4. Pass 4: retirement for `DELETING`/`DELETED`
Start from Recon's own `CLOSED` and `QUASI_CLOSED` containers and move
them
forward only when SCM explicitly returns `DELETING` or `DELETED`.
## Root Causes Addressed
### 1. DN-report path could not advance beyond `CLOSING`
### 2. Sync used to be add-only for `CLOSED`
### 3. Recon could miss `OPEN` and `QUASI_CLOSED` containers entirely
### 4. Recon never retired stale live states based on SCM deletion progress
### 5. SCM batch API could drop containers when pipeline lookup failed
### 6. Recon add path and SCM state manager were not null-pipeline safe
### 7. Open-container count per pipeline could drift
### 8. `decideSyncAction()` became a real tiered decision
**Current logic:**
1. compare total SCM and Recon container counts
2. if total drift `>` `ozone.recon.scm.container.threshold`:
`FULL_SNAPSHOT`
3. else if total drift `> 0`:
`TARGETED_SYNC`
4. else compare per-state drift for:
- `OPEN`
- `QUASI_CLOSED`
- derived `CLOSED` remainder
5. if any per-state drift exceeds
`ozone.recon.scm.per.state.drift.threshold`:
`TARGETED_SYNC`
6. otherwise:
`NO_ACTION`
### 9. Incremental sync got its own schedule
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14758
## How was this patch tested?
### Tests added or updated
- [`TestReconSCMContainerSyncIntegration.java`]
- [`TestReconStorageContainerSyncHelper.java`]
- [`TestTriggerDBSyncEndpoint.java`]
- [`TestUnhealthyContainersDerbyPerformance.java`]
- [`TestReconContainerHealthSummaryEndToEnd.java`]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
