Arafat Khan created HDDS-16091:
----------------------------------
Summary: Recon fills disk with leaked checkpoints and crashes on
startup when local OM DB is missing
Key: HDDS-16091
URL: https://issues.apache.org/jira/browse/HDDS-16091
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Recon
Affects Versions: 2.3.0
Reporter: Arafat Khan
During long replication runs, Recon's {{/var/lib/hadoop-ozone/recon}} directory
grows unbounded (observed ~501 GB) and fills the disk. This is a regression
from HDDS-13809: the checkpoint cleanup was changed to read the checkpoint
directory location _after_ closing the DB store, but closing sets the store to
{{null}} — so the temporary reinitialization checkpoints (each a full copy of
the OM DB) are never deleted.
Once the disk is cleared, if the local OM snapshot DB is missing while stale
task-status rows still exist, Recon crashes on startup with an NPE:{{{}{}}}
{code:java}
java.lang.NullPointerException: ...ReconOMMetadataManager.getStore() is null at
ReconTaskControllerImpl.cleanTempCheckPointPath(...) {code}
{{}}
The HTTP server starts before the crash, so the process stays up but the
datanode RPC service never starts — datanodes cannot connect and show as DEAD.
*Fix*
# Revert the HDDS-13809 Recon changes so the checkpoint directory is captured
_before_ the store is closed, then deleted — restoring correct cleanup.
# Add null-store guards in {{cleanTempCheckPointPath}} and
{{{}cleanupPreExistingCheckpoints{}}}, and skip startup reinitialization when
the local OM DB store is not yet initialized. The scheduled full-snapshot sync
then downloads a fresh OM DB instead of crashing.
*Note:* The OM-side changes from HDDS-13809 ({{{}OMDBCheckpointServlet{}}},
{{{}QuotaRepairTask{}}}) are correct and are left in place. The Kerberos realm
mismatch ({{{}dn/_HOST@REALM{}}}) seen on the cluster is a separate environment
config issue, tracked separately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
