Siyao Meng created HDDS-17013:
---------------------------------
Summary: Tarball and untar directory of a container import
interrupted by a datanode crash are never removed from the volume
Key: HDDS-17013
URL: https://issues.apache.org/jira/browse/HDDS-17013
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-16-remove-interrupted-import-leftovers-at-startup.patch,
TestBugCR16StaleImportTarball.java
h3. Mechanism
A pushed container is written by {{SendContainerRequestHandler}} to
{{<volume>/tmp/container-copy/container-<id>-<random UUID>.tar}}, then untarred
into {{container-copy/<id>}} by {{TarContainerPacker.unpackContainerData}},
called from {{KeyValueContainer.importContainerData}}. The tarball is deleted
only in process ({{onError}}, the failure path of {{onCompleted}}, the
{{finally}} of {{ContainerImporter.importContainer}}); the untar directory is
deleted by the failure path of {{KeyValueContainer.importContainerData}} or
moved into place on success. Nothing else lists or cleans {{container-copy}}:
the volume initialization ({{HddsVolume.createTmpDirs}}) cleans only
directories under {{hdds/<clusterID>/tmp}}, container loading at startup does
not look at it, and a graceful shutdown does not either. A later import of the
same container to the same volume replaces the untar directory, but because the
tarball name contains a random UUID it never replaces the old tarball.
Released versions are affected as well (checked 2.2.1): there the pull
replication path also writes {{container-<id>-<UUID>.tar}} into the same
directory.
h3. Trigger
# SCM has a container pushed to a datanode, and the datanode has received part
of it (or is untarring it).
# The datanode process dies before the import finishes (kill, OOM kill or exit
on OOM, power loss).
# The datanode restarts. The leftovers stay.
h3. Impact
Each import in flight at a crash leaves up to about the size of the container
on that volume (more if the untar had started) until an operator deletes it by
hand. The loss accumulates: a datanode that crashes repeatedly while receiving
re-replication, for example during a decommission or after another datanode
failed, adds more on each crash, and nothing in Ozone reports where the space
went. The space is outside the container accounting, so it lowers the free
space the datanode reports (argued from the code, not measured). No data loss.
h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugCR16StaleImportTarball.java]
({{hadoop-hdds/container-service}}) models a datanode lifetime by what startup
does per data volume ({{StorageVolumeUtil.checkVolume}}, then
{{ContainerReader}} as in {{OzoneContainer.buildContainerSet}}). It pushes
three 1MB parts of a container through the real {{SendContainerRequestHandler}}
and abandons the handler without {{onError}} or {{onCompleted}}, as after a
kill. The tarball is still there after a restart, after a failed push of the
same container, and after a volume set shutdown and a second restart. A passing
test means the defect is present.
h3. Patch
[^CR-16-remove-interrupted-import-leftovers-at-startup.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3.
{{ContainerReader}} empties the volume's {{container-copy}} directory before it
loads the containers of the volume at datanode startup (the {{shouldDelete}}
path; the debug CLI does not use it). Nothing can be importing at that point,
since the replication server is started later in {{OzoneContainer.start}}. The
cleanup is not done in {{HddsVolume.createTmpDirs}}, because that runs on every
version handshake with SCM, including after a reregister, when an import may be
in progress. An exception during the cleanup only logs a warning and does not
stop the volume from loading. The cleanup also removes files left in that
directory by older versions after an upgrade.
Covered by {{testLeftoverContainerImportFilesCleared}} in the existing
{{TestContainerReader}}, which also checks that a run without {{shouldDelete}}
leaves the files alone. Without the change it fails for all container test
versions with the directory not empty; with it the replication and container
loading suites ({{TestSendContainerRequestHandler}}, {{TestContainerImporter}},
{{TestGrpcReplicationService}}, {{TestContainerReader}},
{{TestOzoneContainer}}, 139 tests) pass and checkstyle is clean. With the patch
the reproduction fails because the tarball is gone after the first restart.
HDDS-8162 (under HDDS-8161) would move {{container-copy}} under
{{hdds/<clusterID>/tmp}}, which HDDS-8161 proposes to clean on restart; this
patch does not move the directory and does not conflict with that.
Found by code review of the datanode volume and per-disk RocksDB lifecycle
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
