Chu Cheng Li created HDDS-16319:
-----------------------------------
Summary: Support server side key copy that shares blocks instead
of rewriting the data
Key: HDDS-16319
URL: https://issues.apache.org/jira/browse/HDDS-16319
Project: Apache Ozone
Issue Type: Improvement
Reporter: Chu Cheng Li
S3 CopyObject in Ozone reads every byte of the source key from the datanodes
through the S3 Gateway and writes it back as a new key, so copy latency is
proportional to object size. {{ObjectEndpoint.copyObject()}} opens an
{{OzoneInputStream}} on the source and streams it into a newly created key;
there is no server side copy RPC in OM, and {{ozone sh key cp}} is a client
side stream copy for the same reason.
This proposes an OM {{CopyKey}} request that creates the destination key from
the source key's committed block list, so a copy moves no data.
h3. Why this is not just a new request
Committed blocks are owned by exactly one key and there is no reference
counting in OM or SCM. Deleting or overwriting either key would move the shared
blocks to deletedTable, and KeyDeletingService would hand them to SCM, silently
breaking the surviving key: a datanode asked to delete an already deleted block
succeeds quietly. The one existing protection,
{{OMKeyRequest.filterOutBlocksStillInUse}}, only compares against the new
version of the same key.
h3. Proposed design
* an optional {{sharedBlockGroupId}} on {{KeyInfo}}, zero for every existing
key so nothing needs backfilling. A non-zero value is the objectID of the
lineage root.
* a {{sharedBlockGroupTable}} column family holding one sharer count per copy
lineage, not one row per block, so copying a key with ten thousand blocks costs
a single row.
* KeyDeletingService consults that count at the single point where deletedTable
entries become SCM block deletions. While another sharer is alive the blocks
are withheld from the SCM call and only the metadata is purged, reusing the
path empty keys already take.
* the count drop rides the same Ratis transaction as the purge, and is carried
by the final batch on purpose: losing it after the rows were purged leaks
blocks, which an audit can reclaim, whereas decrementing first and then failing
to purge would let a retry decrement twice and release blocks a live key still
uses.
No block release producer has to change, because the tag travels with OmKeyInfo
into deletedTable through {{OmUtils.prepareKeyForDelete}}. Every failure
direction degrades to a leak rather than to data loss, which is the main reason
for preferring this over a per-block reference count that every producer would
have to maintain correctly.
h3. When a copy must still rewrite the data
Encrypted keys (block data is ciphertext under the source key's own DEK and
IV), GDPR enforced keys (erasure destroys the key's own secret), a replication
config change (the read path picks the EC or Ratis stream from the key's own
config), an hsync active source, and UploadPartCopy, which cannot express a
byte range as whole object sharing.
h3. Measurements
On a three datanode MiniOzoneCluster, medians over five iterations: CopyKey
costs 6.7 to 9.2 ms at 1, 16, 64 and 256 MiB, with no trend across the size
range, because it is one OM Ratis transaction regardless of how much data the
key holds. The read and rewrite copy grows from 117 ms to 11.1 s over the same
range. The ratios overstate a real cluster, whose byte copy beats the 18 to 23
MiB/s a single host manages; projecting from a realistic 100 MiB/s still gives
roughly 320x at 256 MiB and 1280x at 1 GiB.
h3. Relation to HDDS-569
HDDS-569 proposed the same thing in 2018 and states the copy "in ozone world,
this is just a metadata change", but its two attached patches define proto
messages only, and the block reclaim problem above was never addressed. This
issue carries the design and an implementation.
h3. Status
The attached pull request is a proof of concept, scoped to OBJECT_STORE
buckets, same bucket, no destination overwrite. Required before it can merge:
OMLayoutFeature gating, without which an older OM treats the tag as an unknown
proto field and reclaims shared blocks normally. Also outstanding: cross-bucket
copy, destination overwrite, FSO, OzoneManagerVersion negotiation, an "ozone
repair om" count audit, Recon awareness, and S3 Gateway wiring with fallback.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
