[
https://issues.apache.org/jira/browse/HDDS-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16949:
------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> BackgroundService stops permanently when getTasks() throws a RuntimeException
> -----------------------------------------------------------------------------
>
> Key: HDDS-16949
> URL: https://issues.apache.org/jira/browse/HDDS-16949
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> A background service silently stops running forever if getTasks() throws an
> unchecked exception.
> {{BackgroundService.PeriodicalTask#run}} calls {{getTasks()}} outside of any
> try/catch. It runs via
> {{ScheduledThreadPoolExecutor#scheduleWithFixedDelay}}, which suppresses all
> subsequent executions when a run throws, and nothing is logged.
> h3. Observed
> {{TestSnapshotDeletingServiceIntegrationTest#testSnapshotWithFSO}} timed out
> waiting for {{snap1}} to be deep cleaned. The log shows, right before
> DirectoryDeletingService went silent:
> {code}
> ERROR om.SnapshotChainManager
> (SnapshotChainManager.java:hasPreviousGlobalSnapshot(516)) - No snapshot
> found in snapshot chain for provided snapshotId: b0ef2d42-...
> {code}
> {{DirectoryDeletingService#getTasks}} walks
> {{SnapshotChainManager#iterator(true)}} without a lock. A concurrent snapshot
> purge removes a snapshot from the chain and {{hasPreviousGlobalSnapshot}}
> throws {{NoSuchElementException}} from {{Iterator#next}}. The exception
> escapes {{PeriodicalTask#run}}, the periodic task is cancelled, and
> DirectoryDeletingService never runs again on that OM (the thread dump shows
> its scheduled queue empty, while KeyDeletingService still has a pending run).
> In production this would permanently stop directory deletion until the OM
> restarts.
> h3. Fix
> Catch {{RuntimeException}} from {{getTasks()}} in
> {{BackgroundService.PeriodicalTask#run}}, log it, and retry at the next
> interval, so a single bad cycle cannot cancel the schedule for any background
> service. Making the snapshot chain iteration itself race safe is left as a
> follow up.
> - https://github.com/apache/ozone/actions/runs/38252733251/job/114829762795
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
