Gerrit-K commented on code in PR #389: URL: https://github.com/apache/polaris/pull/389#discussion_r2014324360
########## polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java: ########## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.core.storage.s3compatible; + +import static org.apache.polaris.core.config.FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS; +import static org.apache.polaris.core.config.PolarisConfiguration.loadConfig; +import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_KEY_ID; +import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_SECRET_KEY; +import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_TOKEN; + +import jakarta.annotation.Nonnull; +import jakarta.ws.rs.NotAuthorizedException; +import java.net.URI; +import java.util.EnumMap; +import java.util.Set; +import org.apache.polaris.core.PolarisDiagnostics; +import org.apache.polaris.core.storage.InMemoryStorageIntegration; +import org.apache.polaris.core.storage.PolarisCredentialProperty; +import org.apache.polaris.core.storage.StorageUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.profiles.ProfileFileSupplier; +import software.amazon.awssdk.services.sts.StsClient; +import software.amazon.awssdk.services.sts.StsClientBuilder; +import software.amazon.awssdk.services.sts.model.AssumeRoleRequest; +import software.amazon.awssdk.services.sts.model.AssumeRoleResponse; + +/** S3 compatible implementation of PolarisStorageIntegration */ +public class S3CompatibleCredentialsStorageIntegration + extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> { + + private static final Logger LOGGER = + LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class); + + public S3CompatibleCredentialsStorageIntegration() { + super(S3CompatibleCredentialsStorageIntegration.class.getName()); + } + + @Override + public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds( + @Nonnull PolarisDiagnostics diagnostics, + @Nonnull S3CompatibleStorageConfigurationInfo storageConfig, + boolean allowListOperation, + @Nonnull Set<String> allowedReadLocations, + @Nonnull Set<String> allowedWriteLocations) { + + String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId()); + String caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey()); + + EnumMap<PolarisCredentialProperty, String> propertiesMap = + new EnumMap<>(PolarisCredentialProperty.class); + propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint()); Review Comment: tl;dr: I believe the endpoint is currently not set when the `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` feature is activated, but I think it should be (or it should be configurable at the catalog level again). Long story: I tested this MR successfully in an earlier stage (back in early January, before the Quarkus migration) and now wanted to upgrade our PoC setup to the most recent state of this MR. Our setup involves a Ceph S3 backend that doesn't have the STS configured, which means: we need to be able to adjust the `s3.endpoint` property, but can't use subscoped credentials. This path doesn't seem to be possible anymore. Previously, we had `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` disabled (because we weren't aware of it) but `skipCredentialSubscopingIndirection` (on the catalog) enabled. The latter seems to have been removed in one of the updates, so STS is always enforced if not skipped globally. If `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` is set, the `getSubscopedCreds` function is not even called by the `FileIOUtil`, so the `S3FileIO` is initialized without region, endpoint etc., which causes it to fall back to environment variables and eventually failing. For our case, we definitely need to set the endpoint for our Ceph catalog, but also don't want to globally disable STS and/or setting global `AWS_*` variables just to fix the `S3FileIO`, because we want to have AWS S3 and Ceph S3 catalogs in the same Polaris instance. Am I missing something in configuring this or does this not work anymore? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
