Siyao Meng created HDDS-16865:
---------------------------------
Summary: OM deletes its pre-install DB backup before the installed
Ratis checkpoint is synced, so a power loss can leave om.db unopenable
Key: HDDS-16865
URL: https://issues.apache.org/jira/browse/HDDS-16865
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-2-sync-checkpoint-before-replacing-om-db.patch,
TestBugMC2CheckpointNotDurable.java
h3. Mechanism
A follower OM that installs a Ratis snapshot writes every checkpoint file
without fsync: {{RDBSnapshotProvider.untarContentsOfTarball}} extracts the
tarball with Hadoop {{FileUtil.unTar}} (GNU tar),
{{InodeMetadataRocksDBCheckpoint}} adds hard links, and
{{OzoneManager.replaceOMDBWithCheckpoint}} renames the old DB items into
{{om.db.backup.*}} and the checkpoint items into the metadata dir.
{{OzoneManager.installCheckpoint}} then reloads the DB and deletes the backup
with {{FileUtils.deleteFully(dbBackup)}}. Nothing on this path syncs the new
files or the directories; the only syncs are the ones RocksDB issues for its
own new MANIFEST, OPTIONS and directory when the DB is reopened.
With delayed allocation (ext4 and XFS defaults) the metadata of the new files
and the renames are forced to the journal by the RocksDB directory fsync on
reload, and the unlink of the backup follows within seconds (the next journal
commit, or any later fsync on the file system), while the SST data stays in the
page cache until writeback, about 30 seconds later with default
{{vm.dirty_expire_centisecs}}. A power loss or host crash in that window leaves
the installed SST files that were not yet written back (all 22 in the test) at
0 bytes, and one such file is enough for RocksDB to refuse to open {{om.db}}.
There is no backup to fall back to.
h3. Trigger
# A follower OM falls behind the leader's Ratis log purge, so Ratis triggers
install snapshot (or a new OM bootstraps).
# The follower downloads, untars and installs the checkpoint and logs "Install
Checkpoint is finished".
# The host loses power, or the kernel crashes, before the installed files are
written back. A process crash alone does not lose the data.
h3. Impact
On restart RocksDB refuses to open {{om.db}} ("Sst file size mismatch between
expected N and file system 0"), and the old DB is gone, so that OM cannot start
until an operator wipes its metadata and lets it install a checkpoint from the
leader again. The other OMs keep serving, so this costs one OM's availability,
not cluster data. The full OM startup on the damaged directory was not run; it
is argued from {{OmMetadataManagerImpl.loadDB}} using the same RocksDB open
that fails here. All releases with OM HA are affected: the install path dates
from HDDS-1649 (0.5.0), and the release tags from 0.5.0 to 2.2.1 delete the
backup without any sync (code reading, only the commit above was run).
SCM's install path ({{HAUtils.replaceDBWithCheckpoint}} and
{{SCMHAManagerImpl.installCheckpoint}}) has the same gap and is not changed
here.
h3. Reproduction
Two forms, both on the unmodified commit.
[^TestBugMC2CheckpointNotDurable.java] ({{hadoop-ozone/integration-test}})
starts a 3 OM mini cluster with one OM down, writes past the log purge gap and
starts that OM, so the install runs through the normal Ratis download, untar
and swap path. Right after the install finishes it checks that the backup is
gone and runs {{filefrag -v}} on the installed SST files. On XFS it failed with
"Backup of the old DB is deleted, but 22 of 22 installed SST files have data
that is not on disk yet". It needs Linux, {{filefrag}} and ext4 or XFS, and a
{{sync}} by another process right after the install can make it pass.
A real power cut was run separately: the same install with the mini cluster
data dir on a loop device behind dm-flakey, cutting writes 1 second after the
install finished (suspend with {{--nolockfs}}, load a {{drop_writes}} table,
kill the JVM, remount), then opening {{om.db}} with rocksdbjni 10.10.1.1. On
ext4 (defaults, {{data=ordered}}) and on XFS (defaults) the remounted {{om.db}}
had 22 of 22 SST files at 0 bytes, no {{om.db.backup.*}} was left, and the open
failed with "Sst file size mismatch between expected 1562 and file system 0".
One earlier XFS run kept its data; that run did not record whether the files
had already been written back before the cut, and the XFS run that did record
it showed 22 of 22 files still with delayed allocation extents.
h3. Patch
[^MC-2-sync-checkpoint-before-replacing-om-db.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
{{installCheckpoint}} now first fsyncs every file and directory of the
checkpoint (once per inode, so hard links are not synced twice), before it
stops any service, and {{moveCheckpointFiles}} fsyncs the metadata dir after
the renames and the marker deletion. Both use Hadoop {{IOUtils.fsync}}, which
also handles directories. If the checkpoint sync fails, the install fails with
nothing stopped or moved; if the metadata dir sync fails, the existing rollback
restores the old DB. The checkpoint sync costs about the data still dirty at
the end of the untar divided by the disk write bandwidth: 2 GB dirty in 250
files took 4.5 s on a SATA SSD, and a second pass over the now clean files took
4 ms. That is bounded by the kernel dirty limit, not by the checkpoint size, so
expect seconds on NVMe, tens of seconds on SATA SSD and minutes on HDD for a
large checkpoint. Because it runs before the services are stopped, it does not
lengthen the time the OM is offline.
Covered by {{testInstallCheckpointSyncsCheckpointBeforeReplacingDb}} in the
existing {{TestOMRatisSnapshots}}: it fails if the install proceeds when the
checkpoint sync fails, if the old DB is not restored when the metadata dir sync
fails, if any checkpoint inode is not synced, or if the metadata dir is not
synced after the checkpoint {{om.db}} was renamed into it. Without the change
it fails because the install proceeds when the checkpoint sync fails. With it
{{TestOMRatisSnapshots}}, {{TestOMRatisSnapshotTransfer}} and
{{TestOMInstallSnapshotDuringBootstrapping}} (11 tests) pass and checkstyle is
clean. With the patch the reproduction passes, and the same dm-flakey power cut
1 second after the install left all 22 SST files intact and {{om.db}} opening
on both ext4 and XFS.
Found by TLA+ model checking and code review of the OM Ratis snapshot
(checkpoint) install paths under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
