Siyao Meng created HDDS-16949:
---------------------------------
Summary: BackgroundService stops permanently when getTasks()
throws a RuntimeException
Key: HDDS-16949
URL: https://issues.apache.org/jira/browse/HDDS-16949
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Manager
Reporter: Siyao Meng
Assignee: Siyao Meng
A background service silently stops running forever if getTasks() throws an
unchecked exception.
{{BackgroundService.PeriodicalTask#run}} calls {{getTasks()}} outside of any
try/catch. It runs via {{ScheduledThreadPoolExecutor#scheduleWithFixedDelay}},
which suppresses all subsequent executions when a run throws, and nothing is
logged.
h3. Observed
{{TestSnapshotDeletingServiceIntegrationTest#testSnapshotWithFSO}} timed out
waiting for {{snap1}} to be deep cleaned. The log shows, right before
DirectoryDeletingService went silent:
{code}
ERROR om.SnapshotChainManager
(SnapshotChainManager.java:hasPreviousGlobalSnapshot(516)) - No snapshot found
in snapshot chain for provided snapshotId: b0ef2d42-...
{code}
{{DirectoryDeletingService#getTasks}} walks
{{SnapshotChainManager#iterator(true)}} without a lock. A concurrent snapshot
purge removes a snapshot from the chain and {{hasPreviousGlobalSnapshot}}
throws {{NoSuchElementException}} from {{Iterator#next}}. The exception escapes
{{PeriodicalTask#run}}, the periodic task is cancelled, and
DirectoryDeletingService never runs again on that OM (the thread dump shows its
scheduled queue empty, while KeyDeletingService still has a pending run). In
production this would permanently stop directory deletion until the OM restarts.
h3. Fix
Catch {{RuntimeException}} from {{getTasks()}} in
{{BackgroundService.PeriodicalTask#run}}, log it, and retry at the next
interval, so a single bad cycle cannot cancel the schedule for any background
service. Making the snapshot chain iteration itself race safe is left as a
follow up.
- https://github.com/apache/ozone/actions/runs/38252733251/job/114829762795
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
