[
https://issues.apache.org/jira/browse/HDDS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sadanand Shenoy updated HDDS-14651:
-----------------------------------
Description:
Add validation at the end of OM checkpoint transfer so the follower can verify
that:
# All files sent by the leader were received.
2. File contents match checksums (e.g. md5sum)
Current behavior: The leader streams the checkpoint tarball and writes a
completion flag. The follower only checks for this flag and does not validate
file count or integrity.
changes:
* Leader: Write a manifest file (file list + per-file checksums) into the
tarball before the completion flag when validation is enabled
* Follower: After untarring, validate that all manifest entries exist and
their checksums match before treating the transfer as complete
* Make validation optional via a config flag (e.g.
ozone.om.ratis.snapshot.transfer.validation.enabled)
was:
Add validation at the end of OM checkpoint transfer so the follower can verify
that:
# All files sent by the leader were received
# File contents match checksums (e.g. md5sum)
Current behavior: The leader streams the checkpoint tarball and writes a
completion flag. The follower only checks for this flag and does not validate
file count or integrity.Proposed changes:
* Leader: Write a manifest file (file list + per-file checksums) into the
tarball before the completion flag when validation is enabled
* Follower: After untarring, validate that all manifest entries exist and
their checksums match before treating the transfer as complete
* Make validation optional via a config flag (e.g.
ozone.om.ratis.snapshot.transfer.validation.enabled)
> Add optional manifest and checksum validation for OM checkpoint transfer.
> -------------------------------------------------------------------------
>
> Key: HDDS-14651
> URL: https://issues.apache.org/jira/browse/HDDS-14651
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Sadanand Shenoy
> Assignee: Sadanand Shenoy
> Priority: Major
>
> Add validation at the end of OM checkpoint transfer so the follower can
> verify that:
> # All files sent by the leader were received.
> 2. File contents match checksums (e.g. md5sum)
> Current behavior: The leader streams the checkpoint tarball and writes a
> completion flag. The follower only checks for this flag and does not validate
> file count or integrity.
> changes:
> * Leader: Write a manifest file (file list + per-file checksums) into the
> tarball before the completion flag when validation is enabled
> * Follower: After untarring, validate that all manifest entries exist and
> their checksums match before treating the transfer as complete
> * Make validation optional via a config flag (e.g.
> ozone.om.ratis.snapshot.transfer.validation.enabled)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
