[
https://issues.apache.org/jira/browse/HADOOP-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816235#comment-17816235
]
ASF GitHub Bot commented on HADOOP-19050:
-----------------------------------------
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1484857819
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AccessGrantConfiguration.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.AbstractHadoopTestBase;
+import org.junit.Assert;
+import org.junit.Test;
+
+import software.amazon.awssdk.awscore.AwsClient;
+import
software.amazon.awssdk.s3accessgrants.plugin.S3AccessGrantsIdentityProvider;
+import software.amazon.awssdk.services.s3.S3BaseClientBuilder;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+
+import static org.apache.hadoop.fs.s3a.Constants.AWS_S3_ACCESS_GRANTS_ENABLED;
+
+
+/**
+ * Test S3 Access Grants configurations.
+ */
+public class TestS3AccessGrantConfiguration extends AbstractHadoopTestBase {
+ /**
+ * This credential provider will be attached to any client
+ * that has been configured with the S3 Access Grants plugin.
+ * {@link
software.amazon.awssdk.s3accessgrants.plugin.S3AccessGrantsPlugin}.
+ */
+ public static final String
S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS =
+ S3AccessGrantsIdentityProvider.class.getName();
+
+ @Test
+ public void testS3AccessGrantsEnabled() throws IOException,
URISyntaxException {
+ // Feature is explicitly enabled
+ AwsClient s3AsyncClient = getAwsClient(createConfig(true), true);
+ Assert.assertEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3AsyncClient));
+
+ AwsClient s3Client = getAwsClient(createConfig(true), false);
+ Assert.assertEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3Client));
+ }
+
+ @Test
+ public void testS3AccessGrantsDisabled() throws IOException,
URISyntaxException {
+ // Disabled by default
+ AwsClient s3AsyncDefaultClient = getAwsClient(new Configuration(),
true);
+ Assert.assertNotEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3AsyncDefaultClient));
+
+ AwsClient s3DefaultClient = getAwsClient(new Configuration(), true);
+ Assert.assertNotEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3DefaultClient));
+
+ // Disabled if explicitly set
+ AwsClient s3AsyncExplicitlyDisabledClient =
getAwsClient(createConfig(false), true);
+ Assert.assertNotEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3AsyncExplicitlyDisabledClient));
+
+ AwsClient s3ExplicitlyDisabledClient =
getAwsClient(createConfig(false), true);
+ Assert.assertNotEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3ExplicitlyDisabledClient));
+ }
+
+ private Configuration createConfig(boolean s3agEnabled) {
Review Comment:
> I think you'll need to do removeBaseAndBucketOverrides here before setting
the value.
I'm not sure about this because I'm starting a new Hadoop Configuration
object itself rather than the `createConfiguration` methods that we use from
the S3ATestUtils. In the end, I don't think it matters - because as long as we
set the S3 Access Grants properties, that's all that matters to us for the
purpose of this test, no?
> and is there a way to check for if the IAM fallback is set on the client?
Unfortunately not :( Did a lot of digging but in short, the plugins are
"applied" to the client. When we apply the S3 Access Grants plugin on the S3
clients, we get the following identity provider set as the Credential Provider
for this client: `S3AccessGrantsIdentityProvider`. And in the case of the
fallback, the fallback flag is only set on the `S3AccessGrantsIdentityProvider`
class but as a private variable that we cannot access.
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AccessGrantConfiguration.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.AbstractHadoopTestBase;
+import org.junit.Assert;
+import org.junit.Test;
+
+import software.amazon.awssdk.awscore.AwsClient;
+import
software.amazon.awssdk.s3accessgrants.plugin.S3AccessGrantsIdentityProvider;
+import software.amazon.awssdk.services.s3.S3BaseClientBuilder;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+
+import static org.apache.hadoop.fs.s3a.Constants.AWS_S3_ACCESS_GRANTS_ENABLED;
+
+
+/**
+ * Test S3 Access Grants configurations.
+ */
+public class TestS3AccessGrantConfiguration extends AbstractHadoopTestBase {
+ /**
+ * This credential provider will be attached to any client
+ * that has been configured with the S3 Access Grants plugin.
+ * {@link
software.amazon.awssdk.s3accessgrants.plugin.S3AccessGrantsPlugin}.
+ */
+ public static final String
S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS =
+ S3AccessGrantsIdentityProvider.class.getName();
+
+ @Test
+ public void testS3AccessGrantsEnabled() throws IOException,
URISyntaxException {
+ // Feature is explicitly enabled
+ AwsClient s3AsyncClient = getAwsClient(createConfig(true), true);
+ Assert.assertEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3AsyncClient));
+
+ AwsClient s3Client = getAwsClient(createConfig(true), false);
+ Assert.assertEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3Client));
+ }
+
+ @Test
+ public void testS3AccessGrantsDisabled() throws IOException,
URISyntaxException {
+ // Disabled by default
+ AwsClient s3AsyncDefaultClient = getAwsClient(new Configuration(),
true);
+ Assert.assertNotEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3AsyncDefaultClient));
+
+ AwsClient s3DefaultClient = getAwsClient(new Configuration(), true);
+ Assert.assertNotEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3DefaultClient));
+
+ // Disabled if explicitly set
+ AwsClient s3AsyncExplicitlyDisabledClient =
getAwsClient(createConfig(false), true);
+ Assert.assertNotEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3AsyncExplicitlyDisabledClient));
+
+ AwsClient s3ExplicitlyDisabledClient =
getAwsClient(createConfig(false), true);
+ Assert.assertNotEquals(
+ S3_ACCESS_GRANTS_EXPECTED_CREDENTIAL_PROVIDER_CLASS,
+ getCredentialProviderName(s3ExplicitlyDisabledClient));
+ }
+
+ private Configuration createConfig(boolean s3agEnabled) {
Review Comment:
> I think you'll need to do removeBaseAndBucketOverrides here before setting
the value.
I'm not sure about this because I'm starting a new Hadoop Configuration
object itself rather than the `createConfiguration` methods that we use from
the S3ATestUtils. In the end, I don't think it matters - because as long as we
set the S3 Access Grants properties, that's all that matters to us for the
purpose of this test, no?
> and is there a way to check for if the IAM fallback is set on the client?
Unfortunately not :( Did a lot of digging but in short, the plugins are
"applied" to the client. When we apply the S3 Access Grants plugin on the S3
clients, we get the following identity provider set as the Credential Provider
for this client: `S3AccessGrantsIdentityProvider`. And in the case of the
fallback, the fallback flag is only set on the `S3AccessGrantsIdentityProvider`
class but as a private variable that we cannot access.
> Add S3 Access Grants Support in S3A
> -----------------------------------
>
> Key: HADOOP-19050
> URL: https://issues.apache.org/jira/browse/HADOOP-19050
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Jason Han
> Assignee: Jason Han
> Priority: Minor
> Labels: pull-request-available
>
> Add support for S3 Access Grants
> (https://aws.amazon.com/s3/features/access-grants/) in S3A.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
