[
https://issues.apache.org/jira/browse/HDDS-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16319:
----------------------------------
Labels: pull-request-available (was: )
> Support server side key copy that shares blocks instead of rewriting the data
> -----------------------------------------------------------------------------
>
> Key: HDDS-16319
> URL: https://issues.apache.org/jira/browse/HDDS-16319
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Chu Cheng Li
> Priority: Major
> Labels: pull-request-available
>
> S3 CopyObject in Ozone reads every byte of the source key from the datanodes
> through the S3 Gateway and writes it back as a new key, so copy latency is
> proportional to object size. {{ObjectEndpoint.copyObject()}} opens an
> {{OzoneInputStream}} on the source and streams it into a newly created key;
> there is no server side copy RPC in OM, and {{ozone sh key cp}} is a client
> side stream copy for the same reason.
> This proposes an OM {{CopyKey}} request that creates the destination key from
> the source key's committed block list, so a copy moves no data.
> h3. Why this is not just a new request
> Committed blocks are owned by exactly one key and there is no reference
> counting in OM or SCM. Deleting or overwriting either key would move the
> shared blocks to deletedTable, and KeyDeletingService would hand them to SCM,
> silently breaking the surviving key: a datanode asked to delete an already
> deleted block succeeds quietly. The one existing protection,
> {{OMKeyRequest.filterOutBlocksStillInUse}}, only compares against the new
> version of the same key.
> h3. Proposed design
> * an optional {{sharedBlockGroupId}} on {{KeyInfo}}, zero for every existing
> key so nothing needs backfilling. A non-zero value is the objectID of the
> lineage root.
> * a {{sharedBlockGroupTable}} column family holding one sharer count per copy
> lineage, not one row per block, so copying a key with ten thousand blocks
> costs a single row.
> * KeyDeletingService consults that count at the single point where
> deletedTable entries become SCM block deletions. While another sharer is
> alive the blocks are withheld from the SCM call and only the metadata is
> purged, reusing the path empty keys already take.
> * the count drop rides the same Ratis transaction as the purge, and is
> carried by the final batch on purpose: losing it after the rows were purged
> leaks blocks, which an audit can reclaim, whereas decrementing first and then
> failing to purge would let a retry decrement twice and release blocks a live
> key still uses.
> No block release producer has to change, because the tag travels with
> OmKeyInfo into deletedTable through {{OmUtils.prepareKeyForDelete}}. Every
> failure direction degrades to a leak rather than to data loss, which is the
> main reason for preferring this over a per-block reference count that every
> producer would have to maintain correctly.
> h3. When a copy must still rewrite the data
> Encrypted keys (block data is ciphertext under the source key's own DEK and
> IV), GDPR enforced keys (erasure destroys the key's own secret), a
> replication config change (the read path picks the EC or Ratis stream from
> the key's own config), an hsync active source, and UploadPartCopy, which
> cannot express a byte range as whole object sharing.
> h3. Measurements
> On a three datanode MiniOzoneCluster, medians over five iterations: CopyKey
> costs 6.7 to 9.2 ms at 1, 16, 64 and 256 MiB, with no trend across the size
> range, because it is one OM Ratis transaction regardless of how much data the
> key holds. The read and rewrite copy grows from 117 ms to 11.1 s over the
> same range. The ratios overstate a real cluster, whose byte copy beats the 18
> to 23 MiB/s a single host manages; projecting from a realistic 100 MiB/s
> still gives roughly 320x at 256 MiB and 1280x at 1 GiB.
> h3. Relation to HDDS-569
> HDDS-569 proposed the same thing in 2018 and states the copy "in ozone world,
> this is just a metadata change", but its two attached patches define proto
> messages only, and the block reclaim problem above was never addressed. This
> issue carries the design and an implementation.
> h3. Status
> The attached pull request is a proof of concept, scoped to OBJECT_STORE
> buckets, same bucket, no destination overwrite. Required before it can merge:
> OMLayoutFeature gating, without which an older OM treats the tag as an
> unknown proto field and reclaims shared blocks normally. Also outstanding:
> cross-bucket copy, destination overwrite, FSO, OzoneManagerVersion
> negotiation, an "ozone repair om" count audit, Recon awareness, and S3
> Gateway wiring with fallback.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
