Siyao Meng created HDDS-17065:
---------------------------------
Summary: Recon ContainerSizeCountTask stops for good after one
failed run, freezing the container size distribution until restart
Key: HDDS-17065
URL: https://issues.apache.org/jira/browse/HDDS-17065
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-5-keep-container-size-count-task-running-and-recount-after-failed-run.patch,
TestBugCR5ContainerSizeCountTaskStopsAfterFailedRun.java
h3. Mechanism
{{ContainerSizeCountTask.run}} has its {{catch (Throwable)}} outside the
{{while (canRun())}} loop, and {{ReconScmTask.initializeAndRunTask}} passes
every exception up. When one run fails, the thread logs "Exception in Container
Size Distribution task Thread." at DEBUG and ends. The task's running flag
stays true, so {{isRunning()}} keeps reporting the task as running and
{{start()}} would not start it again; nothing else does either.
What fails in practice is the Recon SQL database: the {{truncate}} in
{{runTask}}, or {{findById}}, {{insert}} or {{update}} in {{writeCountsToDB}},
all of which throw a jOOQ {{DataAccessException}} when the database rejects the
call. Listing the containers is in memory, and errors for a single container
are already caught per container.
Letting the loop simply continue is not enough. {{processContainers}} updates
its in memory {{processedContainers}} before {{writeCountsToDB}}, and that
write is an insert followed by a separate update. After a failed write the next
run computes its delta from the already updated map, so the failed change is
never applied; if the insert went through and the update did not, the table
also keeps half of the change. Either way the counts stay wrong until Recon
restarts, which recounts from scratch.
h3. Trigger
Any failure of the Recon SQL database during one run of the task (default every
60s, {{ozone.recon.task.containercounttask.interval}}), for example an outage
of an external database configured with {{ozone.recon.sql.db.jdbc.url}}, or a
full disk under the embedded Derby database.
h3. Impact
* After the thread has ended, {{GET /api/v1/utilization/containerCount}} and
the "Container Size Distribution" chart on Recon's Insights page keep showing
the counts of the last successful run until Recon restarts. If the failed run
was the first one after startup, they show nothing, or the rows the previous
Recon process left, since the table persists across restarts. The task logs the
failure only at DEBUG. In a real database outage {{ReconTaskStatusUpdater}}
also logs an ERROR "Failed to update table for task" without the cause, but
nothing reports that the thread has ended. {{/api/v1/task/status}} shows the
task's last run as failed, with a timestamp that no longer advances.
* Only this statistic is affected; the endpoint is its only reader. SCM,
datanodes and data are not affected.
h3. Reproduction
[^TestBugCR5ContainerSizeCountTaskStopsAfterFailedRun.java] goes in
{{hadoop-ozone/recon}} (target path and Maven command in the class comment) and
passes on unmodified source when the bug is present. It runs the real
{{ContainerSizeCountTask}} on Recon's real SQL schema (embedded Derby) and
reads the result through the real {{UtilizationEndpoint}}; the container
manager is a mock and the DAO is a spy of the real DAO whose first write throws
{{DataAccessException}}.
* {{oneFailedRunStopsTheTask}} (200ms interval): the first run fails, then the
cluster gains a second container. After 12 intervals the task had listed the
containers once, the endpoint returned no rows for two containers,
{{isRunning()}} returned true, and the task logged nothing at INFO or above
(the failure is injected at the size count DAO only, so the task status writes
succeed).
* {{retryAfterPartialWriteKeepsWrongCounts}} shows why a plain retry is not
enough: runs are called directly, a container moves from the 2GB bin to the 4GB
bin, the new 4GB row is inserted and the update of the 2GB row fails. After the
next successful run the endpoint still returns one container in each bin for a
single 2.5GB container.
What was not exercised: a real outage of an external database or a full disk;
the failure is injected at the DAO.
h3. Patch
[^CR-5-keep-container-size-count-task-running-and-recount-after-failed-run.patch],
against 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at
14e0611fdd5.
{{ContainerSizeCountTask.run}} now catches a failed run inside the loop, logs
it at ERROR, records the failed run in the task status and waits the interval
before the next run. {{processContainers}} clears {{processedContainers}} when
the run fails, so the next run truncates the table and recounts all containers,
as Recon does at startup. A recount is used instead of rolling back the in
memory state because the table may already hold part of the failed write. The
recount costs what Recon already does on its first run after startup: a
truncate and one row per non empty size bin, on top of the pass over the
container list that every run makes. A reader may see an empty table for that
moment, as after a restart.
Covered by {{TestContainerSizeCountTask.testTaskKeepsRunningAfterFailedRun}}
(the first write fails, the container must still reach the table) and
{{TestContainerSizeCountTask.testCountsRecoverAfterFailedWrite}} (after a
partial write and one more run only the new bin may be counted). Without the
change the first times out and the second sees both bins. With it the size
count, endpoint, SCM sync helper and SCM facade tests pass, checkstyle is
clean, and the attached test no longer shows the bug.
The same death pattern in {{PipelineSyncTask}} is a separate issue with its own
patch; the two patches touch different files and apply together.
Found by code review of the datanode container report and Recon container
tracking paths, as part of the TLA+ verification effort under HDDS-15926, on
commit 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues
and apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-12227 (resolved, lowered this log line to DEBUG), HDDS-10293
(resolved, negative container size, caught per container), HDDS-7632 (resolved,
added the task). The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
