[
https://issues.apache.org/jira/browse/HDDS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Rose updated HDDS-13245:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Container scanner needs to account for deleted blocks when building the
> merkle tree
> -----------------------------------------------------------------------------------
>
> Key: HDDS-13245
> URL: https://issues.apache.org/jira/browse/HDDS-13245
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Aswin Shakil
> Assignee: Ethan Rose
> Priority: Major
> Labels: pull-request-available
>
> Currently the container scanner rebuilds the merkle tree based only on what
> it sees on disk. This will cause the data checksum to change when blocks are
> deleted, which is not desirable. The scanner should also consult the list of
> deleted blocks already stored in the checksum file and add those to the tree
> it generates.
> There are a few parts to this change:
> * Change the persisted list of deleted blocks from a list of IDs to a list
> {{BlockMerkleTree}} protos so that we also have the checksum there to
> reference, instead of trying to find it in the previous tree.
> ** The block checksum used here should be built from the chunk checksums in
> RocksDB, not the block itself, otherwise it might diverge between replicas if
> deleting a corrupted block.
> ** We can clear out the chunk children from the {{BlockMerkleTree}} before
> adding it to this list to save space. They will never be read.
> * When reconciling, if a peer has marked a block as deleted and we have not,
> but we also don't have the block, we need to add that block and its checksum
> to our deleted block list.
> ** This allows checksums to converge for containers that had blocks deleted
> before upgrading to reconciliation.
> * When reconciling, if we have a checksum mismatch with a peer's deleted
> block, log a warning.
> ** If our copy is corrupted, it is expected to be deleted and that will
> resolve the issue.
> ** If our copy is deleted and the checksum doesn't match, we cannot resolve
> the issue.
> *** If the checksum written at the time of delete doesn't match, it means it
> didn't match in RocksDB either, which is already unrecoverable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
