cryptoe opened a new pull request, #20543:
URL: https://github.com/apache/druid/pull/20543
### Description
MSQ durable storage and S3 exports write files with S3 multipart uploads,
through `RetryableS3OutputStream`. `KmsServerSideEncryption` and
`S3ServerSideEncryption` only added encryption settings to single
uploads and copies, not to the request that starts a multipart upload.
Large durable-storage and export files therefore got the bucket's default
encryption instead of the configured `druid.storage.sse.*`.
With `kms`, this meant `keyId` was never sent. Only files small enough for
a single request were encrypted with the configured settings.
This PR adds the missing settings:
- `kms`: adds `aws:kms` and `ssekmsKeyId` to the request that starts a
multipart upload.
- `s3`: adds `AES256` to the request that starts a multipart upload.
- `custom`: adds the customer key to both the start request and each part
upload, because S3 needs the key on every part.
#### Release note
S3 server-side encryption (`druid.storage.sse.type`) now applies to
multipart uploads in MSQ durable storage and S3 exports. Previously, large
files fell back to the bucket's default encryption.
**Upgrade notes:**
- Clusters using `kms` need no special upgrade order. Reading a
KMS-encrypted object needs no extra request headers, so mixed versions and
rollbacks work.
- Before you upgrade, make sure that the KMS key policy grants
`kms:GenerateDataKey` and `kms:Decrypt` to the roles that write these files,
such as MSQ tasks. AWS requires `kms:Decrypt` for KMS-encrypted
multipart uploads. Grant `kms:Decrypt` to everything that reads them,
including Brokers that read MSQ query results and any systems outside Druid
that read exported files.
- With `custom`, large durable-storage objects that older tasks write
during the upgrade may still fail to read, as they already do today. The
failures stop once all services are upgraded.
<hr>
##### Key changed/added classes in this PR
* `KmsServerSideEncryption`
* `S3ServerSideEncryption`
* `CustomServerSideEncryption`
* `ServerSideEncryptionTest`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
