[
https://issues.apache.org/jira/browse/HDDS-16865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16865:
------------------------------
Attachment: MC-2-sync-checkpoint-before-replacing-om-db.patch
> OM deletes its pre-install DB backup before the installed Ratis checkpoint is
> synced, so a power loss can leave om.db unopenable
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16865
> URL: https://issues.apache.org/jira/browse/HDDS-16865
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-2-sync-checkpoint-before-replacing-om-db.patch,
> TestBugMC2CheckpointNotDurable.java
>
>
> h3. Mechanism
> A follower OM that installs a Ratis snapshot writes every checkpoint file
> without fsync: {{RDBSnapshotProvider.untarContentsOfTarball}} extracts the
> tarball with Hadoop {{FileUtil.unTar}} (GNU tar),
> {{InodeMetadataRocksDBCheckpoint}} adds hard links, and
> {{OzoneManager.replaceOMDBWithCheckpoint}} renames the old DB items into
> {{om.db.backup.*}} and the checkpoint items into the metadata dir.
> {{OzoneManager.installCheckpoint}} then reloads the DB and deletes the backup
> with {{FileUtils.deleteFully(dbBackup)}}. Nothing on this path syncs the new
> files or the directories; the only syncs are the ones RocksDB issues for its
> own new MANIFEST, OPTIONS and directory when the DB is reopened.
> With delayed allocation (ext4 and XFS defaults) the metadata of the new files
> and the renames are forced to the journal by the RocksDB directory fsync on
> reload, and the unlink of the backup follows within seconds (the next journal
> commit, or any later fsync on the file system), while the SST data stays in
> the page cache until writeback, about 30 seconds later with default
> {{vm.dirty_expire_centisecs}}. A power loss or host crash in that window
> leaves the installed SST files that were not yet written back (all 22 in the
> test) at 0 bytes, and one such file is enough for RocksDB to refuse to open
> {{om.db}}. There is no backup to fall back to.
> h3. Trigger
> # A follower OM falls behind the leader's Ratis log purge, so Ratis triggers
> install snapshot (or a new OM bootstraps).
> # The follower downloads, untars and installs the checkpoint and logs
> "Install Checkpoint is finished".
> # The host loses power, or the kernel crashes, before the installed files are
> written back. A process crash alone does not lose the data.
> h3. Impact
> On restart RocksDB refuses to open {{om.db}} ("Sst file size mismatch between
> expected N and file system 0"), and the old DB is gone, so that OM cannot
> start until an operator wipes its metadata and lets it install a checkpoint
> from the leader again. The other OMs keep serving, so this costs one OM's
> availability, not cluster data. The full OM startup on the damaged directory
> was not run; it is argued from {{OmMetadataManagerImpl.loadDB}} using the
> same RocksDB open that fails here. All releases with OM HA are affected: the
> install path dates from HDDS-1649 (0.5.0), and the release tags from 0.5.0 to
> 2.2.1 delete the backup without any sync (code reading, only the commit above
> was run).
> SCM's install path ({{HAUtils.replaceDBWithCheckpoint}} and
> {{SCMHAManagerImpl.installCheckpoint}}) has the same gap and is not changed
> here.
> h3. Reproduction
> Two forms, both on the unmodified commit.
> [^TestBugMC2CheckpointNotDurable.java] ({{hadoop-ozone/integration-test}})
> starts a 3 OM mini cluster with one OM down, writes past the log purge gap
> and starts that OM, so the install runs through the normal Ratis download,
> untar and swap path. Right after the install finishes it checks that the
> backup is gone and runs {{filefrag -v}} on the installed SST files. On XFS it
> failed with "Backup of the old DB is deleted, but 22 of 22 installed SST
> files have data that is not on disk yet". It needs Linux, {{filefrag}} and
> ext4 or XFS, and a {{sync}} by another process right after the install can
> make it pass.
> A real power cut was run separately: the same install with the mini cluster
> data dir on a loop device behind dm-flakey, cutting writes 1 second after the
> install finished (suspend with {{--nolockfs}}, load a {{drop_writes}} table,
> kill the JVM, remount), then opening {{om.db}} with rocksdbjni 10.10.1.1. On
> ext4 (defaults, {{data=ordered}}) and on XFS (defaults) the remounted
> {{om.db}} had 22 of 22 SST files at 0 bytes, no {{om.db.backup.*}} was left,
> and the open failed with "Sst file size mismatch between expected 1562 and
> file system 0". One earlier XFS run kept its data; that run did not record
> whether the files had already been written back before the cut, and the XFS
> run that did record it showed 22 of 22 files still with delayed allocation
> extents.
> h3. Patch
> [^MC-2-sync-checkpoint-before-replacing-om-db.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> {{installCheckpoint}} now first fsyncs every file and directory of the
> checkpoint (once per inode, so hard links are not synced twice), before it
> stops any service, and {{moveCheckpointFiles}} fsyncs the metadata dir after
> the renames and the marker deletion. Both use Hadoop {{IOUtils.fsync}}, which
> also handles directories. If the checkpoint sync fails, the install fails
> with nothing stopped or moved; if the metadata dir sync fails, the existing
> rollback restores the old DB. The checkpoint sync costs about the data still
> dirty at the end of the untar divided by the disk write bandwidth: 2 GB dirty
> in 250 files took 4.5 s on a SATA SSD, and a second pass over the now clean
> files took 4 ms. That is bounded by the kernel dirty limit, not by the
> checkpoint size, so expect seconds on NVMe, tens of seconds on SATA SSD and
> minutes on HDD for a large checkpoint. Because it runs before the services
> are stopped, it does not lengthen the time the OM is offline.
> Covered by {{testInstallCheckpointSyncsCheckpointBeforeReplacingDb}} in the
> existing {{TestOMRatisSnapshots}}: it fails if the install proceeds when the
> checkpoint sync fails, if the old DB is not restored when the metadata dir
> sync fails, if any checkpoint inode is not synced, or if the metadata dir is
> not synced after the checkpoint {{om.db}} was renamed into it. Without the
> change it fails because the install proceeds when the checkpoint sync fails.
> With it {{TestOMRatisSnapshots}}, {{TestOMRatisSnapshotTransfer}} and
> {{TestOMInstallSnapshotDuringBootstrapping}} (11 tests) pass and checkstyle
> is clean. With the patch the reproduction passes, and the same dm-flakey
> power cut 1 second after the install left all 22 SST files intact and
> {{om.db}} opening on both ext4 and XFS.
> Found by TLA+ model checking and code review of the OM Ratis snapshot
> (checkpoint) install paths under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
