[jira] [Updated] (HDDS-14651) Add optional manifest and checksum validation for OM checkpoint transfer.
[ https://issues.apache.org/jira/browse/HDDS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDDS-14651: --- Epic Link: HDDS-15130 (was: HDDS-13747) > Add optional manifest and checksum validation for OM checkpoint transfer. > - > > Key: HDDS-14651 > URL: https://issues.apache.org/jira/browse/HDDS-14651 > Project: Apache Ozone > Issue Type: Bug >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > > Add validation at the end of OM checkpoint transfer so the follower can > verify that: > # All files sent by the leader were received. > 2. File contents match checksums (e.g. md5sum) > Current behavior: The leader streams the checkpoint tarball and writes a > completion flag. The follower only checks for this flag and does not validate > file count or integrity. > changes: > * Leader: Write a manifest file (file list + per-file checksums) into the > tarball before the completion flag when validation is enabled > * Follower: After untarring, validate that all manifest entries exist and > their checksums match before treating the transfer as complete > * Make validation optional via a config flag (e.g. > ozone.om.ratis.snapshot.transfer.validation.enabled) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14651) Add optional manifest and checksum validation for OM checkpoint transfer.
[ https://issues.apache.org/jira/browse/HDDS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDDS-14651: --- Epic Link: HDDS-13747 > Add optional manifest and checksum validation for OM checkpoint transfer. > - > > Key: HDDS-14651 > URL: https://issues.apache.org/jira/browse/HDDS-14651 > Project: Apache Ozone > Issue Type: Bug >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > > Add validation at the end of OM checkpoint transfer so the follower can > verify that: > # All files sent by the leader were received. > 2. File contents match checksums (e.g. md5sum) > Current behavior: The leader streams the checkpoint tarball and writes a > completion flag. The follower only checks for this flag and does not validate > file count or integrity. > changes: > * Leader: Write a manifest file (file list + per-file checksums) into the > tarball before the completion flag when validation is enabled > * Follower: After untarring, validate that all manifest entries exist and > their checksums match before treating the transfer as complete > * Make validation optional via a config flag (e.g. > ozone.om.ratis.snapshot.transfer.validation.enabled) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14651) Add optional manifest and checksum validation for OM checkpoint transfer.
[ https://issues.apache.org/jira/browse/HDDS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sadanand Shenoy updated HDDS-14651: --- Description: Add validation at the end of OM checkpoint transfer so the follower can verify that: # All files sent by the leader were received. 2. File contents match checksums (e.g. md5sum) Current behavior: The leader streams the checkpoint tarball and writes a completion flag. The follower only checks for this flag and does not validate file count or integrity. changes: * Leader: Write a manifest file (file list + per-file checksums) into the tarball before the completion flag when validation is enabled * Follower: After untarring, validate that all manifest entries exist and their checksums match before treating the transfer as complete * Make validation optional via a config flag (e.g. ozone.om.ratis.snapshot.transfer.validation.enabled) was: Add validation at the end of OM checkpoint transfer so the follower can verify that: # All files sent by the leader were received # File contents match checksums (e.g. md5sum) Current behavior: The leader streams the checkpoint tarball and writes a completion flag. The follower only checks for this flag and does not validate file count or integrity.Proposed changes: * Leader: Write a manifest file (file list + per-file checksums) into the tarball before the completion flag when validation is enabled * Follower: After untarring, validate that all manifest entries exist and their checksums match before treating the transfer as complete * Make validation optional via a config flag (e.g. ozone.om.ratis.snapshot.transfer.validation.enabled) > Add optional manifest and checksum validation for OM checkpoint transfer. > - > > Key: HDDS-14651 > URL: https://issues.apache.org/jira/browse/HDDS-14651 > Project: Apache Ozone > Issue Type: Bug >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > > Add validation at the end of OM checkpoint transfer so the follower can > verify that: > # All files sent by the leader were received. > 2. File contents match checksums (e.g. md5sum) > Current behavior: The leader streams the checkpoint tarball and writes a > completion flag. The follower only checks for this flag and does not validate > file count or integrity. > changes: > * Leader: Write a manifest file (file list + per-file checksums) into the > tarball before the completion flag when validation is enabled > * Follower: After untarring, validate that all manifest entries exist and > their checksums match before treating the transfer as complete > * Make validation optional via a config flag (e.g. > ozone.om.ratis.snapshot.transfer.validation.enabled) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
