[
https://issues.apache.org/jira/browse/HDDS-16091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arafat Khan updated HDDS-16091:
-------------------------------
Description:
During long replication runs, Recon's {{/var/lib/hadoop-ozone/recon}} directory
grows unbounded (observed ~501 GB) and fills the disk. This is a regression
from HDDS-13809: the checkpoint cleanup was changed to read the checkpoint
directory location _after_ closing the DB store, but closing sets the store to
{{null}} — so the temporary reinitialization checkpoints (each a full copy of
the OM DB) are never deleted.
Once the disk is cleared, if the local OM snapshot DB is missing while stale
task-status rows still exist, Recon crashes on startup with an NPE -
{code:java}
java.lang.NullPointerException: ...ReconOMMetadataManager.getStore() is null at
ReconTaskControllerImpl.cleanTempCheckPointPath(...) {code}
*Fix*
# Revert the HDDS-13809 Recon changes so the checkpoint directory is captured
_before_ the store is closed, then deleted — restoring correct cleanup.
# Add null-store guards in {{cleanTempCheckPointPath}} and
{{{}cleanupPreExistingCheckpoints{}}}, and skip startup reinitialization when
the local OM DB store is not yet initialized. The scheduled full-snapshot sync
then downloads a fresh OM DB instead of crashing.
was:
During long replication runs, Recon's {{/var/lib/hadoop-ozone/recon}} directory
grows unbounded (observed ~501 GB) and fills the disk. This is a regression
from HDDS-13809: the checkpoint cleanup was changed to read the checkpoint
directory location _after_ closing the DB store, but closing sets the store to
{{null}} — so the temporary reinitialization checkpoints (each a full copy of
the OM DB) are never deleted.
Once the disk is cleared, if the local OM snapshot DB is missing while stale
task-status rows still exist, Recon crashes on startup with an NPE -
{code:java}
java.lang.NullPointerException: ...ReconOMMetadataManager.getStore() is null at
ReconTaskControllerImpl.cleanTempCheckPointPath(...) {code}
The HTTP server starts before the crash, so the process stays up but the
datanode RPC service never starts — datanodes cannot connect and show as DEAD.
*Fix*
# Revert the HDDS-13809 Recon changes so the checkpoint directory is captured
_before_ the store is closed, then deleted — restoring correct cleanup.
# Add null-store guards in {{cleanTempCheckPointPath}} and
{{{}cleanupPreExistingCheckpoints{}}}, and skip startup reinitialization when
the local OM DB store is not yet initialized. The scheduled full-snapshot sync
then downloads a fresh OM DB instead of crashing.
> Recon fills disk with leaked checkpoints and crashes on startup when local OM
> DB is missing
> -------------------------------------------------------------------------------------------
>
> Key: HDDS-16091
> URL: https://issues.apache.org/jira/browse/HDDS-16091
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Affects Versions: 2.3.0
> Reporter: Arafat Khan
> Assignee: Arafat Khan
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> During long replication runs, Recon's {{/var/lib/hadoop-ozone/recon}}
> directory grows unbounded (observed ~501 GB) and fills the disk. This is a
> regression from HDDS-13809: the checkpoint cleanup was changed to read the
> checkpoint directory location _after_ closing the DB store, but closing sets
> the store to {{null}} — so the temporary reinitialization checkpoints (each a
> full copy of the OM DB) are never deleted.
> Once the disk is cleared, if the local OM snapshot DB is missing while stale
> task-status rows still exist, Recon crashes on startup with an NPE -
> {code:java}
> java.lang.NullPointerException: ...ReconOMMetadataManager.getStore() is null
> at ReconTaskControllerImpl.cleanTempCheckPointPath(...) {code}
>
> *Fix*
> # Revert the HDDS-13809 Recon changes so the checkpoint directory is
> captured _before_ the store is closed, then deleted — restoring correct
> cleanup.
> # Add null-store guards in {{cleanTempCheckPointPath}} and
> {{{}cleanupPreExistingCheckpoints{}}}, and skip startup reinitialization when
> the local OM DB store is not yet initialized. The scheduled full-snapshot
> sync then downloads a fresh OM DB instead of crashing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
