Geqi Hu created HDFS-17963:
------------------------------
Summary: Add an offline command to compact excessive finalized
edit-log segments
Key: HDFS-17963
URL: https://issues.apache.org/jira/browse/HDFS-17963
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode, tools
Reporter: Geqi Hu
Healthy HDFS clusters with regularly completed checkpoints normally do not
accumulate an excessive number of finalized edit-log segments. However,
long-lived idle or test clusters may continue rolling edit logs while
checkpointing is disabled, misconfigured, or failing.
I have encountered this situation repeatedly while handling HDFS recovery
incidents with strict recovery-time objectives.
In one anonymized case from an HDFS-compatible distribution:
* 113,340 finalized edit-log segments had accumulated;
* their combined size was only 7,583,956 bytes;
* 107,382 segments (94.74%) were 42-byte segments containing only
OP_START_LOG_SEGMENT and OP_END_LOG_SEGMENT;
* only 5,958 segment files contained non-marker operations; and
* the required replay range was txid 806689 through 1045052.
Two NameNodes replayed the sequence at nearly identical rates of approximately
704 and 705 txids/hour. At txid 827760, only about 8.8% had been processed,
with an estimated 308 hours (12.8 days) remaining.
This suggests that repeated segment opening, initialization, validation, and
closing dominated the recovery time rather than edit-log byte size or
business-operation replay.
Deleting the marker-only files is not safe because their transaction IDs are
part of the continuous history and valid operations are interspersed among
them. Fixing checkpointing prevents future accumulation but does not reduce an
existing backlog when the NameNode cannot complete startup.
I propose adding an explicitly offline and experimental command to rewrite a
continuous sequence of finalized edit-log segments into fewer output segments.
The command should:
* validate filename ranges and decoded transaction IDs;
* reject gaps, overlaps, duplicate ranges, corruption, and incompatible layout
versions;
* preserve transaction IDs, operation order, and operation contents;
* write only to a separate output directory; and
* re-read and validate all generated output before reporting success.
It would not process in-progress segments, modify input files, access a running
NameNode or JournalNode, replace live metadata, renumber transactions, or
manage service startup and rollback.
I implemented an independent prototype. For the case above, it rewrote the
113,340 segments into one output file, produced identical source/output
operation-stream SHA-256 digests, and the NameNode successfully completed
startup with the validated result in a controlled recovery test.
HDFS-2982 addressed repeated directory scans during startup. This proposal
addresses the fixed per-file overhead of processing an already accumulated
set of tens or hundreds of thousands of finalized segments.
Would offline compaction be an acceptable recovery mechanism for this
scenario, and would OfflineEditsViewer be the appropriate integration point?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
