[
https://issues.apache.org/jira/browse/HDDS-17389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17389:
------------------------------
Attachment: TestBugCR1ExportDirSweep.java
> Recon export cleanup deletes every directory and .tar file in
> ozone.recon.export.directory, not only files created by exports
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17389
> URL: https://issues.apache.org/jira/browse/HDDS-17389
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-1-export-startup-cleanup-only-owned-entries.patch,
> TestBugCR1ExportDirSweep.java
>
>
> h3. Mechanism
> The {{ExportJobManager}} constructor removes leftovers of earlier exports
> from {{ozone.recon.export.directory}}. It lists the directory and calls
> {{FileUtils.deleteQuietly}} (recursive) on every subdirectory and on every
> file ending in {{.tar}}, without looking at the name. The manager itself only
> creates {{<jobId>}} working directories (a random UUID) and
> {{export_<state>_<millis>.tar}} files, so everything else it deletes was put
> there by someone else.
> The configured value is used as is. Nothing checks that the directory is
> dedicated to exports or differs from {{ozone.recon.db.dir}}, and neither
> {{ozone-default.xml}} nor {{ReconServerConfigKeys}} says that its contents
> are removed. SCM's container ID export ({{ExportFileManager}}) clears its
> export directory on startup more narrowly: it removes only {{export_<UUID>}}
> directories and {{*.tar.gz.tmp}} files.
> h3. Trigger
> # Set {{ozone.recon.export.directory}} to a directory that holds other data,
> for example the Recon DB directory itself or a shared scratch directory.
> # Start Recon. The cleanup runs when the {{ExportJobManager}} singleton is
> created, which happens once per Recon process on first use of the containers
> API (argued from the Guice binding, not run in a Recon server).
> The default ({{ozone.recon.export.directory}} empty, resolved to
> {{<ozone.recon.db.dir>/exports}}) is not affected.
> h3. Impact
> * With the export directory equal to {{ozone.recon.db.dir}}: Recon's
> container key DB, OM snapshot DB, Recon SCM DB and Derby SQL DB directories
> are deleted, and so is {{recon/}} with Recon's storage {{current/VERSION}}
> file (run against stand-in directories with those names, see reproduction).
> Because the sweep runs on the first containers API request, Recon has these
> databases open at that point, so they are deleted under a running process
> (argued from code, not run).
> * With a shared directory: other applications' directories and any {{.tar}}
> file (for example backups) are deleted permanently (run). Other files are
> kept.
> * It repeats on every Recon start.
> h3. Reproduction
> [^TestBugCR1ExportDirSweep.java] ({{hadoop-ozone/recon}}, package
> {{org.apache.hadoop.ozone.recon.api}}) builds {{ExportJobManager}} through
> its public constructor with real configuration keys, as Guice does; only
> {{ContainerHealthSchemaManager}}, which the cleanup does not use, is mocked.
> The first test points the export directory at the Recon DB directory holding
> {{recon-container-key.db_...}}, {{om.snapshot.db_...}}, {{recon-scm.db}},
> {{ozone_recon_derby.db}} and {{recon/current/VERSION}}; after construction
> the directory is empty. The second uses a shared directory with
> {{other-app-cache/}}, {{operator-backup-2026-10-01.tar}} and {{notes.txt}};
> only {{notes.txt}} is left. Both tests pass on unmodified source,
> deterministic; a passing test means the defect is present.
> h3. Patch
> [^CR-1-export-startup-cleanup-only-owned-entries.patch], against
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
> 9c3e5477ffb.
> The cleanup now removes a directory only if its name is a valid UUID
> ({{UUIDUtil.isValidUuidString}}, as in SCM's {{ExportFileManager}}) and a
> file only if its name has the form {{export_<state>_<millis>.tar}}, the names
> {{submitJob}} and {{executeExport}} create. The state part accepts any
> letters, because {{submitJob}} lower cases it with the JVM default locale
> (under a Turkish locale {{MISSING}} gets a dotless i). Leftovers of an
> interrupted export are still removed and other files are kept. A directory
> whose name is a UUID is still removed, because the job manager names its own
> working directories that way. In the default directory, which only holds
> export files, the behaviour is unchanged.
> Covered in the existing {{TestExportJobManager}}: the new
> {{startupCleanupKeepsEntriesNotCreatedByExports}} checks that {{backup.tar}},
> {{export_backup.tar}} and a DB directory survive, and
> {{startupCleanupRemovesLeftoverTarsAndDirs}} now uses the names the manager
> really creates (it used {{stale_export.tar}} and {{stale_job_id/}} before),
> including a tar named with the Turkish lower case of {{MISSING}}, and still
> expects them removed. The test builds that name with an explicit locale and
> does not change the JVM default. Without the change the new test fails
> because {{backup.tar}} is deleted. With it {{TestExportJobManager}},
> {{TestExportJob}} and {{TestContainerEndpoint}} (54 tests) pass and
> checkstyle is clean. With the patch both tests of the reproduction fail,
> because all entries are kept.
> Found by code review of the Recon unhealthy container export job manager, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
