[
https://issues.apache.org/jira/browse/HDDS-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16318:
----------------------------------
Labels: pull-request-available (was: )
> Support metadata-only self-copy with metadata directive REPLACE
> ---------------------------------------------------------------
>
> Key: HDDS-16318
> URL: https://issues.apache.org/jira/browse/HDDS-16318
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Manager, S3
> Reporter: Chu Cheng Li
> Priority: Major
> Labels: pull-request-available
>
> S3 CopyObject with the same source and destination key plus
> {{x-amz-metadata-directive: REPLACE}} is the mechanism AWS documents for
> updating an object's metadata in place. Today the S3 Gateway serves it by
> streaming every byte of the object from the datanodes through the gateway and
> writing a brand new key ({{ObjectEndpoint#copyObject}}), which is pure waste
> for a metadata-only change.
> h3. Proposal
> Add a new OM write request {{SetObjectMetadata}} (OBS and FSO variants) that
> replaces the custom metadata and the tag set of an existing key under the
> bucket lock.
> * System-managed metadata entries (ETag, GDPR flag/secret/algorithm, hsync
> client id) are preserved server-side. ETag preservation is required by AWS
> semantics: the ETag reflects only changes to the object's contents, not its
> metadata.
> * {{keyLocationVersions}}, {{dataSize}}, replication config and encryption
> info are untouched.
> * {{modificationTime}} is updated because AWS CopyObject updates
> LastModified. This deliberately differs from PutObjectTagging.
> * Tags are replaced in the same atomic request, so
> {{x-amz-tagging-directive}} keeps working for self-copies.
> The S3 Gateway serves a self-copy with directive REPLACE through this
> request, and falls back silently to the existing byte-copy path when the OM
> does not support the API or the cluster is not finalized (new
> {{OzoneManagerVersion}} value plus a new {{OMLayoutFeature}}). No on-disk
> schema change.
> A new cmdType is needed rather than a flag on PutObjectTagging: any OM from
> version 9 onwards already accepts PutObjectTagging, and {{metadata}} is an
> existing {{KeyArgs}} field, so an old OM would replace the tag set, silently
> ignore the metadata and return OK, telling the client the update succeeded
> when it had not. A distinct cmdType makes an old OM reject the request
> cleanly, which is what the gateway fallback keys off.
> h3. Measured effect
> S3 Gateway side only, in-memory stub (no datanode I/O, no replication, no
> RPC), self-copy with metadata directive REPLACE, median of 15 iterations:
> || object size || full copy (today) || metadata only || speedup ||
> | 1 MiB | 2.36 ms | 0.51 ms | 4.6x |
> | 8 MiB | 12.23 ms | 0.35 ms | 34.9x |
> | 32 MiB | 44.79 ms | 0.35 ms | 128.9x |
> | 64 MiB | 89.81 ms | 0.37 ms | 244.9x |
> Today's cost is linear in object size (about 1.4 ms per MiB of gateway CPU);
> the metadata-only update is constant at about 0.35 ms. These are a lower
> bound: the harness has no datanode I/O or replication, so on a cluster the
> same operation additionally stops reading N bytes from datanodes and writing
> N x replication bytes back. A 64 MiB self-copy avoids about 192 MiB of
> replicated writes.
> Related: HDDS-569 proposed a metadata-only copy in 2018.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
