[
https://issues.apache.org/jira/browse/HADOOP-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17402177#comment-17402177
]
Steve Loughran commented on HADOOP-17855:
-----------------------------------------
thinking more about this.
# we could have some plugin point which returned the encryption settings for
each path being written to, would be used when creating a request (i.e in
RequestFactoryImpl) to choose settings in PUT/initiate MPU, copy. There's some
complexity there related to TransferManager though... copy is going to be
trouble.
# this'd be left completely out of the delegation token info passed into the
cluster. Up to the cluster deployer to deal with this. The default encryption
settings would be passed in this way.
# IT'd be (another) hadoop AbstractService created during initialize(), but
we'd make its serviceStart() operation async, so anything it does (load a
config file, bind to some service) wouldn't block normal initialization...the
config is only needed on the first write call
Like I said before: not on my TODO list.
> S3A: Allow SSE configurations per object path
> ---------------------------------------------
>
> Key: HADOOP-17855
> URL: https://issues.apache.org/jira/browse/HADOOP-17855
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.1
> Reporter: Mike Dias
> Priority: Major
>
> Currently, we can map the SSE configurations at bucket level only:
> {code:java}
> <property>
> <name>fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm</name>
> <value>SSE-KMS</value>
> </property>
> <property>
> <name>fs.s3a.bucket.ireland-dev.server-side-encryption.key</name>
>
> <value>arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01</value>
> </property>
> {code}
> But sometimes we want to encrypt data in different paths with different keys
> within the same bucket. For example, a partitioned table might benefit from
> encrypting each partition with a different key when the partition represents
> a customer or a country.
> [S3 already can encrypt using different keys/configurations at the object
> level|https://aws.amazon.com/premiumsupport/knowledge-center/s3-encrypt-specific-folder/],
> so what we need to do on Hadoop is to provide a way to map which key to use.
> One idea could be mapping them in the XML config:
>
> {code:java}
> <property>
> <name>fs.s3a.server-side-encryption.paths</name>
>
> <value>s3://bucket/my_table/country=ireland,s3://bucket/my_table/country=uk,
> s3://bucket/my_table/country=germany</value>
> </property>
> <property>
> <name>fs.s3a.server-side-encryption.path-keys</name>
>
> <value>arn:aws:kms:eu-west-1:90ireland09:key/ireland-key,arn:aws:kms:eu-west-1:980uk0993c:key/uk-key,arn:aws:kms:eu-west-1:98germany089:key/germany-key</value>
> </property>
> {code}
> Or potentially fetch the mappings from the filesystem:
>
> {code:java}
> <property>
> <name>fs.s3a.server-side-encryption.mappings</name>
> <value>s3://bucket/configs/encryption_mappings.json</value>
> </property> {code}
> where encryption_mappings.json could be something like this:
>
> {code:java}
> {
> "path": "s3://bucket/customer_table/customerId=abc123",
> "algorithm": "SSE-KMS",
> "key": "arn:aws:kms:eu-west-1:933993746:key/abc123-key"
> }
> ...
> {
> "path": "s3://bucket/customer_table/customerId=xyx987",
> "algorithm": "SSE-KMS",
> "key": "arn:aws:kms:eu-west-1:933993746:key/xyx987-key"
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
