This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new bb482ff0d0 [#12198] feat(credential): support Tencent Cloud COS
credential vending (#12199)
bb482ff0d0 is described below
commit bb482ff0d0ecf8e48fad95456e35f9dbe0ea20db
Author: Gary Wang <[email protected]>
AuthorDate: Mon Aug 31 17:06:10 2026 +0800
[#12198] feat(credential): support Tencent Cloud COS credential vending
(#12199)
### What changes were proposed in this pull request?
Add credential vending support for the Tencent Cloud COS fileset backend
introduced in #11713. This is a follow-up subtask under #6490.
- Add `COSTokenCredential` to the public credential API, alongside the
existing `S3TokenCredential`, `OSSTokenCredential`,
`GCSTokenCredential`, and `ADLSTokenCredential`.
- Add `cos-token` credential provider implementation (`COSTokenProvider`
+ `COSTokenGenerator`) that calls Tencent Cloud STS `AssumeRole` and
returns temporary credentials scoped to the requested table path via a
generated CAM policy.
- Add lightweight CAM policy model classes (`Policy`, `Statement`,
`Effect`, `Condition`, `StringLike`) shared by the token generator and
covered by unit tests.
- Reuse the existing `cos-secret-key` provider so the two providers
share the same `cos-access-key-id` / `cos-secret-access-key`
configuration surface.
- Register both providers via `META-INF/services` so they are picked up
by `CredentialProviderFactory` and by `CredentialFactory` on the client
side.
- Wire `COSCredentialsProvider` (the Hadoop `AWSCredentialsProvider`) to
consume vended `COSTokenCredential`s, in addition to the static
secret-key credential.
- Update `bundles/tencent` and `bundles/tencent-bundle` to include the
new sources and the Tencent STS SDK dependency.
- Update `CredentialConstants`, `COSCredentialConfig`, and
`COSProperties` to expose the new provider name and configuration keys
(`cos-role-arn`, `cos-region`, `cos-app-id`, `cos-external-id`,
`cos-token-expire-in-secs`).
Tests:
- Unit tests for `COSTokenGenerator`, `COSCredentialConfig`,
`COSCredentialProvider` SPI registration, and `TestCredentialFactory`
round-trip.
- Integration test `GravitinoVirtualFileSystemCOSCredentialIT`
exercising the end-to-end vending path against a real COS bucket through
GVFS.
Documentation:
- Add a `## COS` section to `docs/security/credential-vending.md`,
documenting `cos-token` and `cos-secret-key`, their properties, and the
required CAM trust and permission policies.
- Update `docs/fileset-catalog-with-cos.md` with credential-vending
configuration examples.
### Why are the changes needed?
#11713 added COS as a fileset storage backend but only supported the
static access-key path. Without credential vending, every client that
loads a fileset receives long-lived, coarse-grained Tencent Cloud
credentials, which is inconsistent with how Gravitino already handles
S3, OSS, GCS, and ADLS. This PR closes that gap so COS fileset
deployments can hand out temporary, table-scoped STS credentials.
### Does this PR introduce _any_ user-facing change?
Yes. Two new credential providers, `cos-token` and `cos-secret-key`, are
now available for catalogs using the COS storage backend, together with
the associated `cos-role-arn` / `cos-region` / `cos-app-id` /
`cos-external-id` / `cos-token-expire-in-secs` catalog properties.
Existing catalogs continue to work unchanged.
### How was this patch tested?
- New unit tests under `bundles/tencent` and `catalogs/catalog-common`.
- New integration test `GravitinoVirtualFileSystemCOSCredentialIT` run
locally against a Tencent Cloud COS bucket with an assumable CAM role,
covering both read and write paths through GVFS with vended credentials.
Fix: #12198
Related: #6490, #11713, #11748
---
.../gravitino/credential/COSTokenCredential.java | 147 ++++++++++
.../org.apache.gravitino.credential.Credential | 1 +
bundles/tencent-bundle/build.gradle.kts | 12 +-
bundles/tencent/build.gradle.kts | 2 +
.../cos/credential/COSTokenGenerator.java | 311 +++++++++++++++++++++
.../gravitino/cos/credential/COSTokenProvider.java | 47 ++++
.../gravitino/cos/credential/policy/Condition.java | 53 ++++
.../gravitino/cos/credential/policy/Effect.java | 17 +-
.../gravitino/cos/credential/policy/Policy.java | 78 ++++++
.../gravitino/cos/credential/policy/Statement.java | 119 ++++++++
.../cos/credential/policy/StringLike.java | 59 ++++
.../gravitino/cos/fs/COSCredentialsProvider.java | 24 +-
.../gravitino/cos/fs/COSFileSystemProvider.java | 3 +-
.../java/org/apache/gravitino/cos/fs/COSUtils.java | 14 +-
....apache.gravitino.credential.CredentialProvider | 1 +
.../cos/credential/TestCOSCredentialProvider.java | 29 ++
.../credential/TestCOSCredentialProviderSpi.java | 60 ++++
.../cos/credential/TestCOSTokenGenerator.java | 255 +++++++++++++++++
.../cos/fs/TestCOSCredentialsProvider.java | 42 +++
.../cos/fs/TestCOSFileSystemProvider.java | 18 ++
.../gravitino/credential/CredentialConstants.java | 1 +
.../credential/config/COSCredentialConfig.java | 79 +++++-
.../apache/gravitino/storage/COSProperties.java | 10 +
.../credential/config/TestCOSCredentialConfig.java | 179 ++++++++++++
.../filesystem-hadoop3-runtime/build.gradle.kts | 6 +
.../GravitinoVirtualFileSystemCOSCredentialIT.java | 284 +++++++++++++++++++
.../credential/TestCredentialFactory.java | 48 ++++
docs/fileset-catalog-with-cos.md | 58 +++-
docs/security/credential-vending.md | 85 +++++-
gradle/libs.versions.toml | 2 +
30 files changed, 1997 insertions(+), 47 deletions(-)
diff --git
a/api/src/main/java/org/apache/gravitino/credential/COSTokenCredential.java
b/api/src/main/java/org/apache/gravitino/credential/COSTokenCredential.java
new file mode 100644
index 0000000000..fc306ed68f
--- /dev/null
+++ b/api/src/main/java/org/apache/gravitino/credential/COSTokenCredential.java
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+
+/** Tencent Cloud COS STS token credential. */
+public class COSTokenCredential implements Credential {
+
+ /** COS STS token credential type. */
+ public static final String COS_TOKEN_CREDENTIAL_TYPE = "cos-token";
+ /** The session access key ID (a.k.a. TmpSecretId in Tencent Cloud) used to
access COS data. */
+ public static final String GRAVITINO_COS_SESSION_ACCESS_KEY_ID =
"cos-access-key-id";
+ /**
+ * The session secret access key (a.k.a. TmpSecretKey in Tencent Cloud) used
to access COS data.
+ */
+ public static final String GRAVITINO_COS_SESSION_SECRET_ACCESS_KEY =
"cos-secret-access-key";
+ /** The COS security token (a.k.a. SessionToken in Tencent Cloud). */
+ public static final String GRAVITINO_COS_SESSION_TOKEN =
"cos-security-token";
+
+ private String accessKeyId;
+ private String secretAccessKey;
+ private String securityToken;
+ private long expireTimeInMs;
+
+ /**
+ * Constructs an instance of {@link COSTokenCredential} with session secret
keys and a security
+ * token.
+ *
+ * @param accessKeyId The COS session access key ID.
+ * @param secretAccessKey The COS session secret access key.
+ * @param securityToken The COS security token.
+ * @param expireTimeInMs The COS token expire time in ms.
+ */
+ public COSTokenCredential(
+ String accessKeyId, String secretAccessKey, String securityToken, long
expireTimeInMs) {
+ validate(accessKeyId, secretAccessKey, securityToken, expireTimeInMs);
+ this.accessKeyId = accessKeyId;
+ this.secretAccessKey = secretAccessKey;
+ this.securityToken = securityToken;
+ this.expireTimeInMs = expireTimeInMs;
+ }
+
+ /**
+ * This is the constructor that is used by credential factory to create an
instance of credential
+ * according to the credential information.
+ */
+ public COSTokenCredential() {}
+
+ @Override
+ public String credentialType() {
+ return COS_TOKEN_CREDENTIAL_TYPE;
+ }
+
+ @Override
+ public long expireTimeInMs() {
+ return expireTimeInMs;
+ }
+
+ @Override
+ public Map<String, String> credentialInfo() {
+ return (new ImmutableMap.Builder<String, String>())
+ .put(GRAVITINO_COS_SESSION_ACCESS_KEY_ID, accessKeyId)
+ .put(GRAVITINO_COS_SESSION_SECRET_ACCESS_KEY, secretAccessKey)
+ .put(GRAVITINO_COS_SESSION_TOKEN, securityToken)
+ .build();
+ }
+
+ /**
+ * Initialize the credential with the credential information.
+ *
+ * <p>This method is invoked to deserialize the credential in client side.
+ *
+ * @param credentialInfo The credential information from {@link
#credentialInfo}.
+ * @param expireTimeInMs The expire-time from {@link #expireTimeInMs()}.
+ */
+ @Override
+ public void initialize(Map<String, String> credentialInfo, long
expireTimeInMs) {
+ String accessKeyId =
credentialInfo.get(GRAVITINO_COS_SESSION_ACCESS_KEY_ID);
+ String secretAccessKey =
credentialInfo.get(GRAVITINO_COS_SESSION_SECRET_ACCESS_KEY);
+ String securityToken = credentialInfo.get(GRAVITINO_COS_SESSION_TOKEN);
+ validate(accessKeyId, secretAccessKey, securityToken, expireTimeInMs);
+ this.accessKeyId = accessKeyId;
+ this.secretAccessKey = secretAccessKey;
+ this.securityToken = securityToken;
+ this.expireTimeInMs = expireTimeInMs;
+ }
+
+ /**
+ * Get COS session access key ID.
+ *
+ * @return The COS session access key ID.
+ */
+ public String accessKeyId() {
+ return accessKeyId;
+ }
+
+ /**
+ * Get COS session secret access key.
+ *
+ * @return The COS session secret access key.
+ */
+ public String secretAccessKey() {
+ return secretAccessKey;
+ }
+
+ /**
+ * Get COS security token.
+ *
+ * @return The COS security token.
+ */
+ public String securityToken() {
+ return securityToken;
+ }
+
+ private void validate(
+ String accessKeyId, String secretAccessKey, String securityToken, long
expireTimeInMs) {
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(accessKeyId), "COS access key Id should not be
empty");
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(secretAccessKey), "COS secret access key should
not be empty");
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(securityToken), "COS security token should not
be empty");
+ Preconditions.checkArgument(
+ expireTimeInMs > 0, "The expiration time of COSTokenCredential should
be greater than 0");
+ }
+}
diff --git
a/api/src/main/resources/META-INF/services/org.apache.gravitino.credential.Credential
b/api/src/main/resources/META-INF/services/org.apache.gravitino.credential.Credential
index 3c5672c6cf..5e4f705c5b 100644
---
a/api/src/main/resources/META-INF/services/org.apache.gravitino.credential.Credential
+++
b/api/src/main/resources/META-INF/services/org.apache.gravitino.credential.Credential
@@ -23,6 +23,7 @@ org.apache.gravitino.credential.GCSTokenCredential
org.apache.gravitino.credential.OSSTokenCredential
org.apache.gravitino.credential.OSSSecretKeyCredential
org.apache.gravitino.credential.COSSecretKeyCredential
+org.apache.gravitino.credential.COSTokenCredential
org.apache.gravitino.credential.ADLSTokenCredential
org.apache.gravitino.credential.AzureAccountKeyCredential
org.apache.gravitino.credential.AwsIrsaCredential
diff --git a/bundles/tencent-bundle/build.gradle.kts
b/bundles/tencent-bundle/build.gradle.kts
index 6e421e0a2b..f8c59de939 100644
--- a/bundles/tencent-bundle/build.gradle.kts
+++ b/bundles/tencent-bundle/build.gradle.kts
@@ -33,6 +33,7 @@ dependencies {
implementation(libs.hadoop3.client.runtime)
implementation(libs.hadoop3.cos)
implementation(libs.httpclient)
+ implementation(libs.tencentcloud.sdk.sts)
}
tasks.withType(ShadowJar::class.java) {
@@ -55,13 +56,14 @@ tasks.withType(ShadowJar::class.java) {
exclude(project(":catalogs:hadoop-common"))
}
- // Relocate dependencies to avoid conflicts.
- // hadoop-cos (from com.qcloud.cos:hadoop-cos) bundles the qcloud-cos SDK
and a few common
- // libraries; relocate them under "org.apache.gravitino.tencent.shaded.*"
following the
- // same pattern as the aws/aliyun/azure/gcp bundles.
+ // Relocate transitive utilities to avoid classpath conflicts. Do NOT
relocate `com.qcloud.*`
+ // or `com.tencentcloudapi.*`: `COSCredentialsProvider` extends hadoop-cos's
+ // `AbstractCOSCredentialProvider`, whose `getCredentials()` returns
`com.qcloud.cos.auth.
+ // COSCredentials`; shading would break the override and cause
`AbstractMethodError`.
relocate("com.fasterxml.jackson",
"org.apache.gravitino.tencent.shaded.com.fasterxml.jackson")
relocate("com.google", "org.apache.gravitino.tencent.shaded.com.google")
- relocate("com.qcloud", "org.apache.gravitino.tencent.shaded.com.qcloud")
+ relocate("okhttp3", "org.apache.gravitino.tencent.shaded.okhttp3")
+ relocate("okio", "org.apache.gravitino.tencent.shaded.okio")
relocate("org.apache.commons",
"org.apache.gravitino.tencent.shaded.org.apache.commons")
relocate("org.apache.http",
"org.apache.gravitino.tencent.shaded.org.apache.http")
relocate("org.checkerframework",
"org.apache.gravitino.tencent.shaded.org.checkerframework")
diff --git a/bundles/tencent/build.gradle.kts b/bundles/tencent/build.gradle.kts
index 98aa70196d..734981c91c 100644
--- a/bundles/tencent/build.gradle.kts
+++ b/bundles/tencent/build.gradle.kts
@@ -43,12 +43,14 @@ dependencies {
compileOnly(libs.hadoop3.client.api)
compileOnly(libs.hadoop3.cos)
compileOnly(libs.lombok)
+ compileOnly(libs.tencentcloud.sdk.sts)
testImplementation(libs.hadoop3.client.api)
testImplementation(libs.hadoop3.client.runtime)
testImplementation(libs.hadoop3.cos)
testImplementation(libs.junit.jupiter.api)
testImplementation(libs.junit.jupiter.params)
+ testImplementation(libs.tencentcloud.sdk.sts)
testRuntimeOnly(libs.junit.jupiter.engine)
testRuntimeOnly(libs.bundles.log4j)
}
diff --git
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/COSTokenGenerator.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/COSTokenGenerator.java
new file mode 100644
index 0000000000..0912404819
--- /dev/null
+++
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/COSTokenGenerator.java
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cos.credential;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.tencentcloudapi.common.Credential;
+import com.tencentcloudapi.common.exception.TencentCloudSDKException;
+import com.tencentcloudapi.sts.v20180813.StsClient;
+import com.tencentcloudapi.sts.v20180813.models.AssumeRoleRequest;
+import com.tencentcloudapi.sts.v20180813.models.AssumeRoleResponse;
+import java.io.IOException;
+import java.net.URI;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.gravitino.cos.credential.policy.Effect;
+import org.apache.gravitino.cos.credential.policy.Policy;
+import org.apache.gravitino.cos.credential.policy.Statement;
+import org.apache.gravitino.credential.COSTokenCredential;
+import org.apache.gravitino.credential.CredentialContext;
+import org.apache.gravitino.credential.CredentialGenerator;
+import org.apache.gravitino.credential.PathBasedCredentialContext;
+import org.apache.gravitino.credential.config.COSCredentialConfig;
+
+/** Generates Tencent Cloud COS STS tokens scoped to the requested fileset
paths. */
+public class COSTokenGenerator implements
CredentialGenerator<COSTokenCredential> {
+
+ private static final String POLICY_VERSION = "2.0";
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String accessKeyId;
+ private String secretAccessKey;
+ private String roleArn;
+ private String externalId;
+ private String region;
+ private String appId;
+ private int tokenExpireSecs;
+
+ @Override
+ public void initialize(Map<String, String> properties) {
+ COSCredentialConfig config = new COSCredentialConfig(properties);
+ this.accessKeyId = config.accessKeyID();
+ this.secretAccessKey = config.secretAccessKey();
+ this.roleArn = config.cosRoleArn();
+ this.externalId = config.externalID();
+ this.region = config.region();
+ this.appId = config.appID();
+ this.tokenExpireSecs = config.tokenExpireInSecs();
+ }
+
+ @Override
+ public COSTokenCredential generate(CredentialContext context) throws
Exception {
+ if (!(context instanceof PathBasedCredentialContext)) {
+ return null;
+ }
+
+ PathBasedCredentialContext pathContext = (PathBasedCredentialContext)
context;
+
+ AssumeRoleResponse response =
+ callAssumeRole(
+ pathContext.getReadPaths(), pathContext.getWritePaths(),
pathContext.getUserName());
+
+ com.tencentcloudapi.sts.v20180813.models.Credentials credentials =
response.getCredentials();
+ Long expiredTime = response.getExpiredTime();
+ Preconditions.checkState(
+ credentials != null && expiredTime != null,
+ "Tencent STS AssumeRole returned an incomplete response, requestId:
%s",
+ response.getRequestId());
+ // Tencent STS returns ExpiredTime in seconds; the Credential contract
uses ms.
+ long expireTimeInMs = expiredTime * 1000L;
+ return new COSTokenCredential(
+ credentials.getTmpSecretId(),
+ credentials.getTmpSecretKey(),
+ credentials.getToken(),
+ expireTimeInMs);
+ }
+
+ private AssumeRoleResponse callAssumeRole(
+ Set<String> readLocations, Set<String> writeLocations, String userName)
+ throws TencentCloudSDKException {
+ Credential cred = new Credential(accessKeyId, secretAccessKey);
+ StsClient client = new StsClient(cred, region);
+
+ AssumeRoleRequest request = new AssumeRoleRequest();
+ request.setRoleArn(roleArn);
+ request.setRoleSessionName(getRoleSessionName(userName));
+ request.setDurationSeconds((long) tokenExpireSecs);
+ if (StringUtils.isNotBlank(externalId)) {
+ request.setExternalId(externalId);
+ }
+ request.setPolicy(buildPolicy(readLocations, writeLocations));
+
+ return client.AssumeRole(request);
+ }
+
+ private String buildPolicy(Set<String> readLocations, Set<String>
writeLocations) {
+ Preconditions.checkArgument(
+ !readLocations.isEmpty() || !writeLocations.isEmpty(),
+ "COS token generator requires at least one read or write location");
+ Policy.Builder policyBuilder = Policy.builder().version(POLICY_VERSION);
+
+ Statement.Builder readObjectStatement =
+ Statement.builder()
+ .effect(Effect.ALLOW)
+ .addAction("cos:GetObject")
+ .addAction("cos:HeadObject");
+
+ // LinkedHashMap keeps the emitted statements in a deterministic order.
+ Map<String, Statement.Builder> bucketListStatements = new
LinkedHashMap<>();
+ Map<String, Statement.Builder> bucketMetadataStatements = new
LinkedHashMap<>();
+
+ Stream.concat(readLocations.stream(), writeLocations.stream())
+ .distinct()
+ .forEach(
+ location -> {
+ URI uri = URI.create(location);
+ addObjectResources(readObjectStatement, uri);
+ String bucketResource = getBucketResource(uri);
+ String bucketWildcardResource = getBucketWildcardResource(uri);
+ // CAM requires different resource ARN forms per action:
cos:GetBucket needs
+ // bucket/*, whereas cos:HeadBucket / cos:GetBucketLocation need
bucket/.
+ Statement.Builder listStatement =
+ bucketListStatements.computeIfAbsent(
+ bucketWildcardResource,
+ key ->
+ Statement.builder()
+ .effect(Effect.ALLOW)
+ .addAction("cos:GetBucket")
+ .addResource(key));
+ // CredentialOperationDispatcher merges multiple PathContexts of
the same
+ // credential type into one PathBasedCredentialContext, so
cos:prefix must be
+ // accumulated for every URI here — not just the first.
+ addPrefixPatterns(listStatement, uri);
+ // hadoop-cos calls headBucket during FileSystem.initialize();
without
+ // cos:HeadBucket the vended credentials return 403.
+ bucketMetadataStatements.computeIfAbsent(
+ bucketResource,
+ key ->
+ Statement.builder()
+ .effect(Effect.ALLOW)
+ .addAction("cos:GetBucketLocation")
+ .addAction("cos:HeadBucket")
+ .addResource(key));
+ });
+
+ if (!writeLocations.isEmpty()) {
+ Statement.Builder writeObjectStatement =
+ Statement.builder()
+ .effect(Effect.ALLOW)
+ .addAction("cos:PutObject")
+ .addAction("cos:DeleteObject")
+ .addAction("cos:InitiateMultipartUpload")
+ .addAction("cos:UploadPart")
+ .addAction("cos:ListParts")
+ .addAction("cos:CompleteMultipartUpload")
+ .addAction("cos:AbortMultipartUpload");
+ writeLocations.forEach(
+ location -> addObjectResources(writeObjectStatement,
URI.create(location)));
+ policyBuilder.addStatement(writeObjectStatement.build());
+ }
+
+ if (!bucketListStatements.isEmpty()) {
+ bucketListStatements.values().forEach(builder ->
policyBuilder.addStatement(builder.build()));
+ }
+ bucketMetadataStatements
+ .values()
+ .forEach(builder -> policyBuilder.addStatement(builder.build()));
+
+ policyBuilder.addStatement(readObjectStatement.build());
+
+ try {
+ return objectMapper.writeValueAsString(policyBuilder.build());
+ } catch (JsonProcessingException e) {
+ throw new RuntimeException("Failed to serialize COS session policy", e);
+ }
+ }
+
+ /**
+ * Emits the {@code cos:prefix} condition patterns for the given fileset
location. For non-root
+ * paths this is {@code xxx/} and {@code xxx/*}; the trailing slash is
essential — a bare {@code
+ * xxx*} would also match sibling prefixes like {@code xxx_backup/}. For
bucket-root filesets
+ * ({@code cosn://bucket/} or {@code cosn://bucket}) the path is empty and a
bare {@code *} is
+ * emitted, since COS object keys have no leading slash.
+ */
+ private void addPrefixPatterns(Statement.Builder statement, URI uri) {
+ String prefix = trimLeadingSlash(uri.getPath());
+ if (prefix.isEmpty()) {
+ statement.addStringLikePrefix("*");
+ return;
+ }
+ if (!prefix.endsWith("/")) {
+ prefix = prefix + "/";
+ }
+ statement.addStringLikePrefix(prefix);
+ statement.addStringLikePrefix(prefix + "*");
+ }
+
+ /** Bucket ARN with trailing slash, used for cos:HeadBucket /
cos:GetBucketLocation. */
+ private String getBucketResource(URI uri) {
+ return getResourcePrefix() + getBucketWithAppId(uri) + "/";
+ }
+
+ /** Bucket ARN with trailing wildcard, required by CAM for cos:GetBucket. */
+ private String getBucketWildcardResource(URI uri) {
+ return getResourcePrefix() + getBucketWithAppId(uri) + "/*";
+ }
+
+ /**
+ * Emits both {@code bucket/prefix} (matches the fileset key itself, e.g. a
HEAD on the root) and
+ * {@code bucket/prefix/*} (matches everything under it). CAM treats them as
distinct resources.
+ */
+ private void addObjectResources(Statement.Builder statement, URI uri) {
+ String path = trimLeadingSlash(uri.getPath());
+ String bucketPrefix = getResourcePrefix() + getBucketWithAppId(uri) + "/";
+ String fullPrefix = bucketPrefix + path;
+ String prefixArn =
+ fullPrefix.endsWith("/") ? fullPrefix.substring(0, fullPrefix.length()
- 1) : fullPrefix;
+ statement.addResource(prefixArn);
+ statement.addResource(appendWildcard(fullPrefix));
+ }
+
+ private String getResourcePrefix() {
+ return "qcs::cos:" + region + ":uid/" + appId + ":";
+ }
+
+ /** Bucket ARNs require the {@code -<APPID>} suffix; append it unless
already present. */
+ private String getBucketWithAppId(URI uri) {
+ String bucket = uri.getHost();
+ if (bucket == null) {
+ throw new IllegalArgumentException("COS location is missing bucket: " +
uri);
+ }
+ String suffix = "-" + appId;
+ if (bucket.endsWith(suffix)) {
+ return bucket;
+ }
+ return bucket + suffix;
+ }
+
+ private String trimLeadingSlash(String path) {
+ if (path == null) {
+ return "";
+ }
+ return path.startsWith("/") ? path.substring(1) : path;
+ }
+
+ /** Appends {@code /*} to a path, avoiding double slashes. */
+ private static String appendWildcard(String leftPath) {
+ return leftPath.endsWith("/") ? leftPath + "*" : leftPath + "/*";
+ }
+
+ private String getRoleSessionName(String userName) {
+ String safe = userName == null ? "anonymous" :
userName.replaceAll("[^A-Za-z0-9_=.@\\-]", "_");
+ // Tencent Cloud caps the role session name at 64 characters.
+ String name = "gravitino_" + safe;
+ return name.length() > 64 ? name.substring(0, 64) : name;
+ }
+
+ // Visible for tests.
+ String buildPolicyForTest(Set<String> readLocations, Set<String>
writeLocations) {
+ return buildPolicy(readLocations, writeLocations);
+ }
+
+ // Visible for tests.
+ void initializeForTest(
+ String accessKeyId,
+ String secretAccessKey,
+ String roleArn,
+ String externalId,
+ String region,
+ String appId,
+ int tokenExpireSecs) {
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(region),
+ "COS token generator requires a non-blank region; got '%s'",
+ region);
+ this.accessKeyId = accessKeyId;
+ this.secretAccessKey = secretAccessKey;
+ this.roleArn = roleArn;
+ this.externalId = externalId;
+ this.region = region;
+ this.appId = appId;
+ this.tokenExpireSecs = tokenExpireSecs;
+ }
+
+ @Override
+ public void close() throws IOException {
+ // StsClient has no resources to release.
+ }
+}
diff --git
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/COSTokenProvider.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/COSTokenProvider.java
new file mode 100644
index 0000000000..531f96a38f
--- /dev/null
+++
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/COSTokenProvider.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cos.credential;
+
+import org.apache.gravitino.credential.COSTokenCredential;
+import org.apache.gravitino.credential.CredentialProviderDelegator;
+
+/**
+ * A lightweight credential provider for Tencent Cloud COS. It delegates the
actual credential
+ * generation to {@link COSTokenGenerator}, which is loaded via reflection so
that bundles without
+ * the Tencent Cloud STS SDK on the classpath do not fail at class loading
time.
+ */
+public class COSTokenProvider extends
CredentialProviderDelegator<COSTokenCredential> {
+
+ @Override
+ public boolean supportsScheme(String scheme) {
+ // hadoop-cos exposes the `cosn://` scheme, matching
`COSFileSystemProvider#scheme()`.
+ return "cosn".equalsIgnoreCase(scheme);
+ }
+
+ @Override
+ public String credentialType() {
+ return COSTokenCredential.COS_TOKEN_CREDENTIAL_TYPE;
+ }
+
+ @Override
+ public String getGeneratorClassName() {
+ return "org.apache.gravitino.cos.credential.COSTokenGenerator";
+ }
+}
diff --git
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Condition.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Condition.java
new file mode 100644
index 0000000000..abc2d76b8d
--- /dev/null
+++
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Condition.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cos.credential.policy;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/** Tencent Cloud CAM policy condition operator wrapper. Currently only {@code
string_like}. */
+@JsonInclude(JsonInclude.Include.NON_NULL)
+public class Condition {
+
+ @SuppressWarnings("UnusedVariable") // Read reflectively by Jackson via
@JsonProperty.
+ @JsonProperty("string_like")
+ private StringLike stringLike;
+
+ private Condition(Builder builder) {
+ this.stringLike = builder.stringLike;
+ }
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ public static class Builder {
+ private StringLike stringLike;
+
+ public Builder stringLike(StringLike stringLike) {
+ this.stringLike = stringLike;
+ return this;
+ }
+
+ public Condition build() {
+ return new Condition(this);
+ }
+ }
+}
diff --git
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/COSProperties.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Effect.java
similarity index 52%
copy from
catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/COSProperties.java
copy to
bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Effect.java
index 7326ebc335..6a3f6c2a78 100644
---
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/COSProperties.java
+++
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Effect.java
@@ -16,19 +16,12 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.gravitino.storage;
-// Properties for Tencent Cloud COS.
-public class COSProperties {
+package org.apache.gravitino.cos.credential.policy;
- // The region of Tencent Cloud COS, e.g. "ap-guangzhou".
- public static final String GRAVITINO_COS_REGION = "cos-region";
- // The endpoint of Tencent Cloud COS (optional, normally derived from
region).
- public static final String GRAVITINO_COS_ENDPOINT = "cos-endpoint";
- // The static access key ID (Tencent Cloud SecretId) used to access COS data.
- public static final String GRAVITINO_COS_ACCESS_KEY_ID = "cos-access-key-id";
- // The static secret access key (Tencent Cloud SecretKey) used to access COS
data.
- public static final String GRAVITINO_COS_ACCESS_KEY_SECRET =
"cos-secret-access-key";
+/** CAM policy statement {@code effect} constants. Only {@code allow} is used
today. */
+public class Effect {
+ public static final String ALLOW = "allow";
- private COSProperties() {}
+ private Effect() {}
}
diff --git
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Policy.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Policy.java
new file mode 100644
index 0000000000..0ccc01b691
--- /dev/null
+++
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Policy.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cos.credential.policy;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Tencent Cloud CAM policy document used as the {@code Policy} parameter of
{@code sts:AssumeRole}.
+ * The serialized JSON follows the format described in the official Tencent
Cloud CAM documentation
+ * (e.g. {@code {"version":"2.0","statement":[...]}}).
+ */
+@JsonInclude(JsonInclude.Include.NON_NULL)
+public class Policy {
+
+ @JsonProperty("version")
+ private String version;
+
+ @JsonProperty("statement")
+ private List<Statement> statements;
+
+ private Policy(Builder builder) {
+ this.version = builder.version;
+ this.statements = builder.statements;
+ }
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ public static class Builder {
+ private String version;
+ private final List<Statement> statements = new ArrayList<>();
+
+ public Builder version(String version) {
+ this.version = version;
+ return this;
+ }
+
+ public Builder addStatement(Statement statement) {
+ this.statements.add(statement);
+ return this;
+ }
+
+ public Policy build() {
+ return new Policy(this);
+ }
+ }
+
+ @SuppressWarnings("unused")
+ public String getVersion() {
+ return version;
+ }
+
+ @SuppressWarnings("unused")
+ public List<Statement> getStatements() {
+ return statements;
+ }
+}
diff --git
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Statement.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Statement.java
new file mode 100644
index 0000000000..242010e6ce
--- /dev/null
+++
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/Statement.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cos.credential.policy;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.util.ArrayList;
+import java.util.List;
+
+/** A Tencent Cloud CAM policy statement. */
+@JsonInclude(JsonInclude.Include.NON_NULL)
+public class Statement {
+
+ @JsonProperty("effect")
+ private String effect;
+
+ @JsonProperty("action")
+ private List<String> actions;
+
+ @JsonProperty("resource")
+ private List<String> resources;
+
+ @JsonProperty("condition")
+ private Condition condition;
+
+ private Statement(Builder builder) {
+ this.effect = builder.effect;
+ this.actions = builder.actions;
+ this.resources = builder.resources;
+ this.condition = builder.condition;
+ }
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ public static class Builder {
+ private String effect;
+ private final List<String> actions = new ArrayList<>();
+ private final List<String> resources = new ArrayList<>();
+ private Condition condition;
+ // Lazily accumulates cos:prefix values across calls to
addStringLikePrefix.
+ private StringLike.Builder stringLikeBuilder;
+
+ public Builder effect(String effect) {
+ this.effect = effect;
+ return this;
+ }
+
+ public Builder addAction(String action) {
+ this.actions.add(action);
+ return this;
+ }
+
+ public Builder addResource(String resource) {
+ this.resources.add(resource);
+ return this;
+ }
+
+ public Builder condition(Condition condition) {
+ this.condition = condition;
+ return this;
+ }
+
+ /** Appends a cos:prefix pattern to the statement's string_like condition.
*/
+ public Builder addStringLikePrefix(String prefix) {
+ if (stringLikeBuilder == null) {
+ stringLikeBuilder = StringLike.builder();
+ }
+ stringLikeBuilder.addPrefix(prefix);
+ return this;
+ }
+
+ public Statement build() {
+ // Explicit condition() wins; only auto-assemble from accumulated
prefixes if unset.
+ if (condition == null && stringLikeBuilder != null) {
+ condition =
Condition.builder().stringLike(stringLikeBuilder.build()).build();
+ }
+ return new Statement(this);
+ }
+ }
+
+ @SuppressWarnings("unused")
+ public String getEffect() {
+ return effect;
+ }
+
+ @SuppressWarnings("unused")
+ public List<String> getActions() {
+ return actions;
+ }
+
+ @SuppressWarnings("unused")
+ public List<String> getResources() {
+ return resources;
+ }
+
+ @SuppressWarnings("unused")
+ public Condition getCondition() {
+ return condition;
+ }
+}
diff --git
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/StringLike.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/StringLike.java
new file mode 100644
index 0000000000..7e3febbe27
--- /dev/null
+++
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/policy/StringLike.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cos.credential.policy;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * CAM {@code string_like} condition value. {@code cos:prefix} is the literal
condition key defined
+ * by Tencent Cloud CAM (analogous to AWS IAM {@code s3:prefix}); it must be
serialized verbatim
+ * into the policy JSON.
+ */
+@JsonInclude(JsonInclude.Include.NON_NULL)
+public class StringLike {
+
+ @SuppressWarnings("UnusedVariable") // Read reflectively by Jackson via
@JsonProperty.
+ @JsonProperty("cos:prefix")
+ private List<String> prefix;
+
+ private StringLike(Builder builder) {
+ this.prefix = builder.prefix;
+ }
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ public static class Builder {
+ private final List<String> prefix = new ArrayList<>();
+
+ public Builder addPrefix(String prefix) {
+ this.prefix.add(prefix);
+ return this;
+ }
+
+ public StringLike build() {
+ return new StringLike(this);
+ }
+ }
+}
diff --git
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSCredentialsProvider.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSCredentialsProvider.java
index 038271ffcf..0c72de44c4 100644
---
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSCredentialsProvider.java
+++
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSCredentialsProvider.java
@@ -20,19 +20,21 @@
package org.apache.gravitino.cos.fs;
import com.qcloud.cos.auth.BasicCOSCredentials;
+import com.qcloud.cos.auth.BasicSessionCredentials;
import com.qcloud.cos.auth.COSCredentials;
import java.net.URI;
import org.apache.gravitino.catalog.hadoop.fs.FileSystemUtils;
import
org.apache.gravitino.catalog.hadoop.fs.GravitinoFileSystemCredentialsProvider;
import org.apache.gravitino.credential.COSSecretKeyCredential;
+import org.apache.gravitino.credential.COSTokenCredential;
import org.apache.gravitino.credential.Credential;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.auth.AbstractCOSCredentialProvider;
/**
* Hadoop-COS credential provider that pulls vended credentials out of
Gravitino and feeds them to
- * the underlying {@code com.qcloud.cos.auth.COSCredentialsProvider} contract.
PR-A only handles
- * static secret-key credentials; STS / token credentials will be added by a
follow-up PR.
+ * the underlying {@code com.qcloud.cos.auth.COSCredentialsProvider} contract.
Handles both static
+ * secret-key credentials and dynamic STS session credentials.
*/
public class COSCredentialsProvider extends AbstractCOSCredentialProvider {
@@ -71,11 +73,21 @@ public class COSCredentialsProvider extends
AbstractCOSCredentialProvider {
this.basicCredentials =
new BasicCOSCredentials(
cosSecretKeyCredential.accessKeyId(),
cosSecretKeyCredential.secretAccessKey());
+ } else if (credential instanceof COSTokenCredential) {
+ COSTokenCredential cosTokenCredential = (COSTokenCredential) credential;
+ // hadoop-cos's BasicSessionCredentials models a temporary STS triple
(TmpSecretId,
+ // TmpSecretKey, SessionToken). The companion
BasicSessionCredentials(appId, ak, sk,
+ // token) constructor is only required when callers explicitly pin an
AppId; here the
+ // AppId is already encoded in the bucket name, so the 3-arg form is
sufficient.
+ this.basicCredentials =
+ new BasicSessionCredentials(
+ cosTokenCredential.accessKeyId(),
+ cosTokenCredential.secretAccessKey(),
+ cosTokenCredential.securityToken());
} else {
- // Defensive: COSUtils#getSuitableCredential currently only returns
- // COSSecretKeyCredential, but a follow-up PR will add STS / token
support. Failing fast
- // here avoids silently leaving {@link #basicCredentials} stale (or
null) if a new
- // credential type is wired into the selector without updating this
branch.
+ // Defensive: any new credential type wired into
COSUtils#getSuitableCredential without
+ // a matching branch here would otherwise silently leave {@link
#basicCredentials} stale
+ // (or null). Failing fast surfaces the wiring bug at refresh time.
throw new RuntimeException(
"Unsupported credential type for COS: " +
credential.getClass().getName());
}
diff --git
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSFileSystemProvider.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSFileSystemProvider.java
index cdc430f27b..916d9068cc 100644
---
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSFileSystemProvider.java
+++
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSFileSystemProvider.java
@@ -30,6 +30,7 @@ import
org.apache.gravitino.catalog.hadoop.fs.FileSystemProvider;
import org.apache.gravitino.catalog.hadoop.fs.FileSystemUtils;
import org.apache.gravitino.catalog.hadoop.fs.SupportsCredentialVending;
import org.apache.gravitino.credential.COSSecretKeyCredential;
+import org.apache.gravitino.credential.COSTokenCredential;
import org.apache.gravitino.credential.Credential;
import org.apache.gravitino.storage.COSProperties;
import org.apache.hadoop.conf.Configuration;
@@ -84,7 +85,7 @@ public class COSFileSystemProvider implements
FileSystemProvider, SupportsCreden
public Map<String, String> getFileSystemCredentialConf(Credential[]
credentials) {
Credential credential = COSUtils.getSuitableCredential(credentials);
Map<String, String> result = Maps.newHashMap();
- if (credential instanceof COSSecretKeyCredential) {
+ if (credential instanceof COSSecretKeyCredential || credential instanceof
COSTokenCredential) {
result.put(
CosNConfigKeys.COSN_CREDENTIALS_PROVIDER,
COSCredentialsProvider.class.getCanonicalName());
diff --git
a/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSUtils.java
b/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSUtils.java
index 1a1dfcf664..4550e8a64d 100644
--- a/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSUtils.java
+++ b/bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSUtils.java
@@ -20,19 +20,27 @@
package org.apache.gravitino.cos.fs;
import org.apache.gravitino.credential.COSSecretKeyCredential;
+import org.apache.gravitino.credential.COSTokenCredential;
import org.apache.gravitino.credential.Credential;
public class COSUtils {
/**
- * Get the credential from the credential array. PR-A only ships static
secret-key support; STS
- * token credentials will be added by a follow-up PR, at which point this
helper should mirror
- * {@code OSSUtils#getSuitableCredential} and prefer dynamic over static
credentials.
+ * Get the credential from the credential array. Using dynamic credential
first, if not found,
+ * uses static credential.
*
* @param credentials The credential array.
* @return A credential. Null if not found.
*/
static Credential getSuitableCredential(Credential[] credentials) {
+ // Use dynamic credential if found.
+ for (Credential credential : credentials) {
+ if (credential instanceof COSTokenCredential) {
+ return credential;
+ }
+ }
+
+ // If dynamic credential not found, use the static one.
for (Credential credential : credentials) {
if (credential instanceof COSSecretKeyCredential) {
return credential;
diff --git
a/bundles/tencent/src/main/resources/META-INF/services/org.apache.gravitino.credential.CredentialProvider
b/bundles/tencent/src/main/resources/META-INF/services/org.apache.gravitino.credential.CredentialProvider
index 8877014411..8b0f89bab9 100644
---
a/bundles/tencent/src/main/resources/META-INF/services/org.apache.gravitino.credential.CredentialProvider
+++
b/bundles/tencent/src/main/resources/META-INF/services/org.apache.gravitino.credential.CredentialProvider
@@ -17,3 +17,4 @@
# under the License.
#
org.apache.gravitino.cos.credential.COSSecretKeyProvider
+org.apache.gravitino.cos.credential.COSTokenProvider
diff --git
a/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSCredentialProvider.java
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSCredentialProvider.java
index 1a83c3795e..9a2361619a 100644
---
a/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSCredentialProvider.java
+++
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSCredentialProvider.java
@@ -21,6 +21,7 @@ package org.apache.gravitino.cos.credential;
import com.google.common.collect.ImmutableMap;
import org.apache.gravitino.credential.COSSecretKeyCredential;
+import org.apache.gravitino.credential.COSTokenCredential;
import org.apache.gravitino.credential.Credential;
import org.apache.gravitino.storage.COSProperties;
import org.junit.jupiter.api.Assertions;
@@ -60,4 +61,32 @@ public class TestCOSCredentialProvider {
Assertions.assertEquals("ak", typed.accessKeyId());
Assertions.assertEquals("sk", typed.secretAccessKey());
}
+
+ @Test
+ void testTokenProviderMetadata() {
+ COSTokenProvider provider = new COSTokenProvider();
+ Assertions.assertEquals(
+ COSTokenCredential.COS_TOKEN_CREDENTIAL_TYPE,
provider.credentialType());
+ Assertions.assertTrue(provider.supportsScheme("cosn"));
+ Assertions.assertTrue(provider.supportsScheme("COSN"));
+ Assertions.assertFalse(provider.supportsScheme("oss"));
+ Assertions.assertEquals(
+ "org.apache.gravitino.cos.credential.COSTokenGenerator",
provider.getGeneratorClassName());
+ }
+
+ @Test
+ void testTokenProviderRejectsNonPathContext() {
+ COSTokenProvider provider = new COSTokenProvider();
+ provider.initialize(
+ ImmutableMap.of(
+ COSProperties.GRAVITINO_COS_ACCESS_KEY_ID, "ak",
+ COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET, "sk",
+ COSProperties.GRAVITINO_COS_ROLE_ARN,
"qcs::cam::uin/100:roleName/role",
+ COSProperties.GRAVITINO_COS_APP_ID, "1259000000",
+ COSProperties.GRAVITINO_COS_REGION, "ap-shanghai"));
+ // Pass a non-path CredentialContext (just a username); the generator
should short-circuit and
+ // return null without reaching the STS service.
+ Credential credential = provider.getCredential(() -> "user");
+ Assertions.assertNull(credential);
+ }
}
diff --git
a/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSCredentialProviderSpi.java
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSCredentialProviderSpi.java
new file mode 100644
index 0000000000..43f0707bcc
--- /dev/null
+++
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSCredentialProviderSpi.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cos.credential;
+
+import java.util.HashSet;
+import java.util.ServiceLoader;
+import java.util.Set;
+import org.apache.gravitino.credential.CredentialProvider;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Guards the SPI wiring in {@code
META-INF/services/org.apache.gravitino.credential
+ * .CredentialProvider}. If a future change accidentally drops {@link
COSTokenProvider} (or {@link
+ * COSSecretKeyProvider}) from that file, the Gravitino server would silently
return an empty {@code
+ * credentials: []} response instead of a proper credential. This test fails
loudly at build time so
+ * the regression cannot slip past CI.
+ */
+public class TestCOSCredentialProviderSpi {
+
+ @Test
+ void testCredentialProviderSpiRegistersBothCosProviders() {
+ Set<Class<?>> loaded = new HashSet<>();
+ for (CredentialProvider provider :
ServiceLoader.load(CredentialProvider.class)) {
+ loaded.add(provider.getClass());
+ }
+
+ // We assert containment (not equality) because other bundles on the test
classpath may
+ // register their own providers via the same SPI file.
+ Assertions.assertTrue(
+ loaded.contains(COSSecretKeyProvider.class),
+ "COSSecretKeyProvider not registered via ServiceLoader; check "
+ + "bundles/tencent/src/main/resources/META-INF/services/"
+ + "org.apache.gravitino.credential.CredentialProvider. Loaded
providers: "
+ + loaded);
+ Assertions.assertTrue(
+ loaded.contains(COSTokenProvider.class),
+ "COSTokenProvider not registered via ServiceLoader; missing this line
means "
+ + "cos-token credential vending will silently return empty
credentials at runtime. "
+ + "Loaded providers: "
+ + loaded);
+ }
+}
diff --git
a/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSTokenGenerator.java
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSTokenGenerator.java
new file mode 100644
index 0000000000..b996048415
--- /dev/null
+++
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSTokenGenerator.java
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.cos.credential;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableSet;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class TestCOSTokenGenerator {
+
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private COSTokenGenerator newGenerator() {
+ COSTokenGenerator generator = new COSTokenGenerator();
+ generator.initializeForTest(
+ "ak", "sk", "qcs::cam::uin/100:roleName/role", null, "ap-shanghai",
"1259000000", 3600);
+ return generator;
+ }
+
+ @Test
+ void testPolicyContainsBucketAppIdSuffix() throws Exception {
+ COSTokenGenerator generator = newGenerator();
+ String json =
+ generator.buildPolicyForTest(
+ ImmutableSet.of("cosn://my-bucket/dataset/foo/"),
ImmutableSet.of());
+ JsonNode root = MAPPER.readTree(json);
+ Assertions.assertEquals("2.0", root.get("version").asText());
+ JsonNode statements = root.get("statement");
+ Assertions.assertTrue(statements.isArray() && statements.size() > 0);
+
+ boolean foundBucketAppIdInResource = false;
+ for (JsonNode stmt : statements) {
+ JsonNode resources = stmt.get("resource");
+ if (resources != null && resources.isArray()) {
+ for (JsonNode r : resources) {
+ if (r.asText().contains("my-bucket-1259000000")) {
+ foundBucketAppIdInResource = true;
+ break;
+ }
+ }
+ }
+ }
+ Assertions.assertTrue(
+ foundBucketAppIdInResource,
+ "Resource ARN should append the APPID suffix to the bucket name.
Policy: " + json);
+ }
+
+ @Test
+ void testPolicyOmitsWriteStatementWhenWriteEmpty() throws Exception {
+ COSTokenGenerator generator = newGenerator();
+ String json =
+
generator.buildPolicyForTest(ImmutableSet.of("cosn://my-bucket/read/"),
ImmutableSet.of());
+ Assertions.assertFalse(json.contains("cos:PutObject"), "Policy: " + json);
+ Assertions.assertFalse(json.contains("cos:DeleteObject"), "Policy: " +
json);
+ }
+
+ @Test
+ void testPolicyIncludesWriteStatementWhenWriteSet() throws Exception {
+ COSTokenGenerator generator = newGenerator();
+ String json =
+ generator.buildPolicyForTest(
+ ImmutableSet.of("cosn://my-bucket/read/"),
ImmutableSet.of("cosn://my-bucket/write/"));
+ Assertions.assertTrue(json.contains("cos:PutObject"), "Policy: " + json);
+ Assertions.assertTrue(json.contains("cos:DeleteObject"), "Policy: " +
json);
+ Assertions.assertTrue(json.contains("cos:CompleteMultipartUpload"),
"Policy: " + json);
+ }
+
+ @Test
+ void testPolicyUsesLowerCaseEffectAllow() throws Exception {
+ COSTokenGenerator generator = newGenerator();
+ String json =
+
generator.buildPolicyForTest(ImmutableSet.of("cosn://my-bucket/data/"),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains("\"effect\":\"allow\""), "Policy: " +
json);
+ Assertions.assertFalse(json.contains("\"Effect\":\"Allow\""), "Policy: " +
json);
+ }
+
+ @Test
+ void testPolicyUsesCosPrefixCondition() throws Exception {
+ COSTokenGenerator generator = newGenerator();
+ String json =
+
generator.buildPolicyForTest(ImmutableSet.of("cosn://my-bucket/data/"),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains("string_like"), "Policy: " + json);
+ Assertions.assertTrue(json.contains("cos:prefix"), "Policy: " + json);
+ Assertions.assertTrue(json.contains("\"data/\""), "Policy: " + json);
+ Assertions.assertTrue(json.contains("\"data/*\""), "Policy: " + json);
+ }
+
+ @Test
+ void testBucketWithExistingAppIdSuffixIsKept() throws Exception {
+ COSTokenGenerator generator = newGenerator();
+ String json =
+ generator.buildPolicyForTest(
+ ImmutableSet.of("cosn://already-1259000000/data/"),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains("already-1259000000"), "Policy: " +
json);
+ Assertions.assertFalse(json.contains("already-1259000000-1259000000"),
"Policy: " + json);
+ }
+
+ @Test
+ void testBuildPolicyRejectsEmptyLocations() {
+ COSTokenGenerator generator = newGenerator();
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> generator.buildPolicyForTest(ImmutableSet.of(),
ImmutableSet.of()));
+ }
+
+ @Test
+ void testPolicyAppendsWildcardWhenLocationHasNoTrailingSlash() throws
Exception {
+ // A location without a trailing slash is normalized to end with '/'
before the cos:prefix
+ // patterns are built; a bare 'data*' pattern MUST NOT be emitted,
otherwise sibling
+ // prefixes such as 'data_backup/' would be accidentally covered.
+ COSTokenGenerator generator = newGenerator();
+ String json =
+ generator.buildPolicyForTest(ImmutableSet.of("cosn://my-bucket/data"),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains("data/*"), "Policy: " + json);
+ Assertions.assertTrue(json.contains("\"data/\""), "Policy: " + json);
+ Assertions.assertTrue(json.contains("\"data/*\""), "Policy: " + json);
+ Assertions.assertFalse(json.contains("\"data*\""), "Policy: " + json);
+ }
+
+ @Test
+ void testInitializeForTestRejectsBlankRegion() {
+ // A blank region degrades the resource ARN region segment to '*',
weakening the policy.
+ COSTokenGenerator generator = new COSTokenGenerator();
+ for (String blank : new String[] {null, "", " "}) {
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ generator.initializeForTest(
+ "ak", "sk", "qcs::cam::uin/100:roleName/role", null, blank,
"1259000000", 3600),
+ "Blank region should be rejected: [" + blank + "]");
+ }
+ }
+
+ @Test
+ void testPolicyResourceArnCarriesConfiguredRegion() throws Exception {
+ COSTokenGenerator generator = newGenerator();
+ String json =
+
generator.buildPolicyForTest(ImmutableSet.of("cosn://my-bucket/data/"),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains("qcs::cos:ap-shanghai:"), "Policy: " +
json);
+ Assertions.assertFalse(json.contains("qcs::cos:*:"), "Policy: " + json);
+ }
+
+ @Test
+ void testPolicyIncludesHeadBucketAction() throws Exception {
+ // hadoop-cos calls headBucket during FileSystem.initialize(); without
cos:HeadBucket the
+ // vended credentials return 403.
+ COSTokenGenerator generator = newGenerator();
+ String json =
+
generator.buildPolicyForTest(ImmutableSet.of("cosn://my-bucket/data/"),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains("cos:HeadBucket"), "Policy: " + json);
+ }
+
+ @Test
+ void testGetBucketUsesWildcardResource() throws Exception {
+ // CAM needs bucket/* for cos:GetBucket and bucket/ for cos:HeadBucket /
cos:GetBucketLocation.
+ COSTokenGenerator generator = newGenerator();
+ String json =
+
generator.buildPolicyForTest(ImmutableSet.of("cosn://my-bucket/data/"),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains(":my-bucket-1259000000/*"), "Policy: "
+ json);
+ Assertions.assertTrue(json.contains(":my-bucket-1259000000/\""), "Policy:
" + json);
+ }
+
+ @Test
+ void testPrefixConditionKeepsTrailingSlashBoundary() throws Exception {
+ // Guard against dropping the '/' before '*' (would over-grant siblings)
or a double
+ // slash ('data//*') from double-appending the wildcard.
+ COSTokenGenerator generator = newGenerator();
+ String json =
+
generator.buildPolicyForTest(ImmutableSet.of("cosn://my-bucket/data/"),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains("\"data/*\""), "Policy: " + json);
+ Assertions.assertFalse(json.contains("data//*"), "Policy: " + json);
+ Assertions.assertFalse(json.contains("\"data*\""), "Policy: " + json);
+ }
+
+ @Test
+ void testPrefixConditionDoesNotOverGrantSiblingPaths() throws Exception {
+ // Regression guard: 'orders' and 'orders/' must both normalize to
'orders/' + 'orders/*',
+ // never leaking to sibling prefixes like 'orders_backup/'.
+ COSTokenGenerator generator = newGenerator();
+ for (String location : new String[] {"cosn://my-bucket/orders",
"cosn://my-bucket/orders/"}) {
+ String json = generator.buildPolicyForTest(ImmutableSet.of(location),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains("\"orders/\""), location + " Policy:
" + json);
+ Assertions.assertTrue(json.contains("\"orders/*\""), location + "
Policy: " + json);
+ Assertions.assertFalse(json.contains("\"orders*\""), location + "
Policy: " + json);
+ }
+ }
+
+ @Test
+ void testPrefixConditionForBucketRootLocation() throws Exception {
+ // Bucket-root fileset: cos:prefix "/" would never match real COS keys
(they carry no
+ // leading slash), so both cosn://bucket/ and cosn://bucket must emit a
bare "*".
+ COSTokenGenerator generator = newGenerator();
+ for (String location : new String[] {"cosn://my-bucket/",
"cosn://my-bucket"}) {
+ String json = generator.buildPolicyForTest(ImmutableSet.of(location),
ImmutableSet.of());
+ Assertions.assertTrue(json.contains("\"*\""), location + " Policy: " +
json);
+ Assertions.assertFalse(json.contains("\"/\""), location + " Policy: " +
json);
+ Assertions.assertFalse(json.contains("\"/*\""), location + " Policy: " +
json);
+ }
+ }
+
+ @Test
+ void testWritePolicyIncludesFullMultipartActionSet() throws Exception {
+ // hadoop-cos calls ListParts on UploadPart 409 to reconcile part state;
missing any of
+ // these actions turns a recoverable multipart upload into a hard 403.
+ COSTokenGenerator generator = newGenerator();
+ String json =
+ generator.buildPolicyForTest(ImmutableSet.of(),
ImmutableSet.of("cosn://my-bucket/data/"));
+ for (String action :
+ new String[] {
+ "cos:PutObject",
+ "cos:DeleteObject",
+ "cos:InitiateMultipartUpload",
+ "cos:UploadPart",
+ "cos:ListParts",
+ "cos:CompleteMultipartUpload",
+ "cos:AbortMultipartUpload"
+ }) {
+ Assertions.assertTrue(json.contains(action), "missing " + action + " in
policy: " + json);
+ }
+ }
+
+ @Test
+ void testGetBucketConditionCoversAllPrefixesForSameBucket() throws Exception
{
+ // CredentialOperationDispatcher.mergeContexts collects multiple
PathContexts into one
+ // PathBasedCredentialContext, so the GetBucket statement must authorise
every path.
+ COSTokenGenerator generator = newGenerator();
+ String json =
+ generator.buildPolicyForTest(
+ ImmutableSet.of("cosn://my-bucket/path-a/"),
+ ImmutableSet.of("cosn://my-bucket/path-b/"));
+ Assertions.assertTrue(json.contains("\"path-a/\""), "Policy: " + json);
+ Assertions.assertTrue(json.contains("\"path-a/*\""), "Policy: " + json);
+ Assertions.assertTrue(json.contains("\"path-b/\""), "Policy: " + json);
+ Assertions.assertTrue(json.contains("\"path-b/*\""), "Policy: " + json);
+ }
+}
diff --git
a/bundles/tencent/src/test/java/org/apache/gravitino/cos/fs/TestCOSCredentialsProvider.java
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/fs/TestCOSCredentialsProvider.java
index 76ff4d1d11..3bdf06e814 100644
---
a/bundles/tencent/src/test/java/org/apache/gravitino/cos/fs/TestCOSCredentialsProvider.java
+++
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/fs/TestCOSCredentialsProvider.java
@@ -20,12 +20,14 @@
package org.apache.gravitino.cos.fs;
import com.qcloud.cos.auth.BasicCOSCredentials;
+import com.qcloud.cos.auth.BasicSessionCredentials;
import com.qcloud.cos.auth.COSCredentials;
import java.net.URI;
import java.util.concurrent.atomic.AtomicInteger;
import
org.apache.gravitino.catalog.hadoop.fs.GravitinoFileSystemCredentialsProvider;
import
org.apache.gravitino.catalog.hadoop.fs.InMemoryFileSystemCredentialsProvider;
import org.apache.gravitino.credential.COSSecretKeyCredential;
+import org.apache.gravitino.credential.COSTokenCredential;
import org.apache.gravitino.credential.Credential;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.CosNConfigKeys;
@@ -108,6 +110,46 @@ public class TestCOSCredentialsProvider {
"Expected message about no suitable credential, but got: " +
ex.getMessage());
}
+ @Test
+ void testRefreshWithTokenCredential() {
+ // Token credentials must be exposed as BasicSessionCredentials so that
hadoop-cos signs
+ // requests with the SessionToken header (x-cos-security-token).
+ long expireMs = System.currentTimeMillis() + 60_000L;
+ StubGravitinoFileSystemCredentialsProvider.nextCredentials =
+ new Credential[] {new COSTokenCredential("tmp-ak", "tmp-sk",
"sts-token", expireMs)};
+
+ COSCredentialsProvider provider = new COSCredentialsProvider(COS_URI,
newStubConf());
+
+ COSCredentials credentials = provider.getCredentials();
+
+ Assertions.assertTrue(
+ credentials instanceof BasicSessionCredentials,
+ "Expected BasicSessionCredentials, got: " +
credentials.getClass().getName());
+ BasicSessionCredentials session = (BasicSessionCredentials) credentials;
+ Assertions.assertEquals("tmp-ak", session.getCOSAccessKeyId());
+ Assertions.assertEquals("tmp-sk", session.getCOSSecretKey());
+ Assertions.assertEquals("sts-token", session.getSessionToken());
+ }
+
+ @Test
+ void testTokenCredentialPreferredOverSecretKey() {
+ // When both credential types are present, COSUtils#getSuitableCredential
must pick the
+ // dynamic (token) one. This mirrors the OSS bundle's selection policy.
+ long expireMs = System.currentTimeMillis() + 60_000L;
+ StubGravitinoFileSystemCredentialsProvider.nextCredentials =
+ new Credential[] {
+ new COSSecretKeyCredential("static-ak", "static-sk"),
+ new COSTokenCredential("tmp-ak", "tmp-sk", "sts-token", expireMs)
+ };
+
+ COSCredentialsProvider provider = new COSCredentialsProvider(COS_URI,
newStubConf());
+
+ COSCredentials credentials = provider.getCredentials();
+
+ Assertions.assertTrue(credentials instanceof BasicSessionCredentials);
+ Assertions.assertEquals("tmp-ak", credentials.getCOSAccessKeyId());
+ }
+
@Test
void testGetCredentialsIsCachedForNonExpiringCredential() {
// COSSecretKeyCredential#expireTimeInMs() returns 0, so expirationTime
stays at
diff --git
a/bundles/tencent/src/test/java/org/apache/gravitino/cos/fs/TestCOSFileSystemProvider.java
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/fs/TestCOSFileSystemProvider.java
index 399e782754..55bece3d20 100644
---
a/bundles/tencent/src/test/java/org/apache/gravitino/cos/fs/TestCOSFileSystemProvider.java
+++
b/bundles/tencent/src/test/java/org/apache/gravitino/cos/fs/TestCOSFileSystemProvider.java
@@ -22,6 +22,7 @@ package org.apache.gravitino.cos.fs;
import java.util.HashMap;
import java.util.Map;
import org.apache.gravitino.credential.COSSecretKeyCredential;
+import org.apache.gravitino.credential.COSTokenCredential;
import org.apache.gravitino.credential.Credential;
import org.apache.gravitino.storage.COSProperties;
import org.apache.hadoop.fs.CosNConfigKeys;
@@ -65,6 +66,23 @@ public class TestCOSFileSystemProvider {
conf.get(CosNConfigKeys.COSN_CREDENTIALS_PROVIDER));
}
+ @Test
+ void testGetFileSystemCredentialConfWithTokenCredential() {
+ COSFileSystemProvider provider = new COSFileSystemProvider();
+ long expireMs = System.currentTimeMillis() + 60_000L;
+ Credential[] credentials =
+ new Credential[] {new COSTokenCredential("tmp-ak", "tmp-sk",
"sts-token", expireMs)};
+
+ Map<String, String> conf =
provider.getFileSystemCredentialConf(credentials);
+
+ // Token credentials must wire the same COSCredentialsProvider; the actual
session token
+ // travels through Gravitino's GravitinoFileSystemCredentialsProvider
rather than via
+ // hadoop-cos config keys.
+ Assertions.assertEquals(
+ COSCredentialsProvider.class.getCanonicalName(),
+ conf.get(CosNConfigKeys.COSN_CREDENTIALS_PROVIDER));
+ }
+
@Test
void testGetFileSystemCredentialConfWithEmptyCredentials() {
COSFileSystemProvider provider = new COSFileSystemProvider();
diff --git
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/CredentialConstants.java
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/CredentialConstants.java
index ea10725b7b..7de528526b 100644
---
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/CredentialConstants.java
+++
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/CredentialConstants.java
@@ -41,6 +41,7 @@ public class CredentialConstants {
"s3-credential-list-location-prefix";
public static final String OSS_TOKEN_EXPIRE_IN_SECS =
"oss-token-expire-in-secs";
+ public static final String COS_TOKEN_EXPIRE_IN_SECS =
"cos-token-expire-in-secs";
public static final String ADLS_TOKEN_EXPIRE_IN_SECS =
"adls-token-expire-in-secs";
/** The HTTP header used to get the credential from fileset location */
diff --git
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/config/COSCredentialConfig.java
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/config/COSCredentialConfig.java
index ab9b2b0faf..338f686cf0 100644
---
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/config/COSCredentialConfig.java
+++
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/config/COSCredentialConfig.java
@@ -26,12 +26,12 @@ import org.apache.gravitino.Config;
import org.apache.gravitino.config.ConfigBuilder;
import org.apache.gravitino.config.ConfigConstants;
import org.apache.gravitino.config.ConfigEntry;
+import org.apache.gravitino.credential.CredentialConstants;
import org.apache.gravitino.storage.COSProperties;
/**
- * Slim credential config for Tencent Cloud COS, covering only the static
secret-key path. STS /
- * token-related entries (role arn, token expire) will be added by a follow-up
PR that introduces
- * the dynamic credential vending support.
+ * Credential config for Tencent Cloud COS. Covers both the static secret-key
path and the dynamic
+ * STS (Security Token Service) path used by credential vending.
*/
public class COSCredentialConfig extends Config {
@@ -40,6 +40,7 @@ public class COSCredentialConfig extends Config {
.doc("The region of the Tencent Cloud COS service")
.version(ConfigConstants.VERSION_2_0_0)
.stringConf()
+ .checkValue(StringUtils::isNotBlank,
ConfigConstants.NOT_BLANK_ERROR_MSG)
.create();
public static final ConfigEntry<String> COS_ACCESS_KEY_ID =
@@ -58,11 +59,65 @@ public class COSCredentialConfig extends Config {
.checkValue(StringUtils::isNotBlank,
ConfigConstants.NOT_BLANK_ERROR_MSG)
.create();
+ public static final ConfigEntry<String> COS_ROLE_ARN =
+ new ConfigBuilder(COSProperties.GRAVITINO_COS_ROLE_ARN)
+ .doc(
+ "The Cloud Access Management (CAM) role ARN that the server
assumes when issuing"
+ + " STS temporary credentials")
+ .version(ConfigConstants.VERSION_2_0_0)
+ .stringConf()
+ .checkValue(StringUtils::isNotBlank,
ConfigConstants.NOT_BLANK_ERROR_MSG)
+ .create();
+
+ public static final ConfigEntry<String> COS_EXTERNAL_ID =
+ new ConfigBuilder(COSProperties.GRAVITINO_COS_EXTERNAL_ID)
+ .doc("Optional external ID for cross-account assume-role")
+ .version(ConfigConstants.VERSION_2_0_0)
+ .stringConf()
+ .create();
+
+ public static final ConfigEntry<String> COS_APP_ID =
+ new ConfigBuilder(COSProperties.GRAVITINO_COS_APP_ID)
+ .doc("The Tencent Cloud APPID that owns the COS buckets, used to
build resource ARNs")
+ .version(ConfigConstants.VERSION_2_0_0)
+ .stringConf()
+ .checkValue(StringUtils::isNotBlank,
ConfigConstants.NOT_BLANK_ERROR_MSG)
+ .create();
+
+ /**
+ * The maximum {@code DurationSeconds} accepted by Tencent Cloud CAM {@code
AssumeRole}, in
+ * seconds. Sourced from the {@code tencentcloud-sdk-java-sts} v3.1.1239
model {@code
+ * AssumeRoleRequest#DurationSeconds}: default 7200 seconds, maximum 43200
seconds (12 hours). If
+ * a future SDK/API bump raises this ceiling, update the constant below and
the accompanying error
+ * message together.
+ */
+ private static final int COS_TOKEN_EXPIRE_IN_SECS_MAX = 43200;
+
+ private static final String COS_TOKEN_EXPIRE_IN_SECS_ERROR_MSG =
+ "cos-token-expire-in-secs must be a positive integer no greater than "
+ + COS_TOKEN_EXPIRE_IN_SECS_MAX
+ + " seconds (Tencent Cloud CAM AssumeRole DurationSeconds hard
limit)";
+
+ public static final ConfigEntry<Integer> COS_TOKEN_EXPIRE_IN_SECS =
+ new ConfigBuilder(CredentialConstants.COS_TOKEN_EXPIRE_IN_SECS)
+ .doc(
+ "COS STS token expire time in seconds. Must be within Tencent
Cloud CAM"
+ + " AssumeRole limits: (0, 43200]. The effective upper bound
may be further"
+ + " reduced by the CAM role's MaxSessionDuration setting;
the STS API rejects"
+ + " values above the role's limit at call time.")
+ .version(ConfigConstants.VERSION_2_0_0)
+ .intConf()
+ .checkValue(
+ v -> v != null && v > 0 && v <= COS_TOKEN_EXPIRE_IN_SECS_MAX,
+ COS_TOKEN_EXPIRE_IN_SECS_ERROR_MSG)
+ .createWithDefault(3600);
+
public COSCredentialConfig(Map<String, String> properties) {
super(false);
loadFromMap(properties, k -> true);
}
+ @NotNull
public String region() {
return this.get(COS_REGION);
}
@@ -76,4 +131,22 @@ public class COSCredentialConfig extends Config {
public String secretAccessKey() {
return this.get(COS_SECRET_ACCESS_KEY);
}
+
+ @NotNull
+ public String cosRoleArn() {
+ return this.get(COS_ROLE_ARN);
+ }
+
+ public String externalID() {
+ return this.get(COS_EXTERNAL_ID);
+ }
+
+ @NotNull
+ public String appID() {
+ return this.get(COS_APP_ID);
+ }
+
+ public Integer tokenExpireInSecs() {
+ return this.get(COS_TOKEN_EXPIRE_IN_SECS);
+ }
}
diff --git
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/COSProperties.java
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/COSProperties.java
index 7326ebc335..d25dcd3a1f 100644
---
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/COSProperties.java
+++
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/COSProperties.java
@@ -30,5 +30,15 @@ public class COSProperties {
// The static secret access key (Tencent Cloud SecretKey) used to access COS
data.
public static final String GRAVITINO_COS_ACCESS_KEY_SECRET =
"cos-secret-access-key";
+ // The CAM role ARN that the server assumes when issuing temporary STS
credentials.
+ // Format: qcs::cam::uin/<owner-uin>:roleName/<role-name>
+ public static final String GRAVITINO_COS_ROLE_ARN = "cos-role-arn";
+ // Optional external ID associated with the CAM role; required only when the
role is configured
+ // with a third-party external ID for cross-account assume-role.
+ public static final String GRAVITINO_COS_EXTERNAL_ID = "cos-external-id";
+ // The Tencent Cloud APPID that owns the COS buckets, used to build resource
ARNs in the STS
+ // session policy. Required when STS credential vending is enabled.
+ public static final String GRAVITINO_COS_APP_ID = "cos-app-id";
+
private COSProperties() {}
}
diff --git
a/catalogs/catalog-common/src/test/java/org/apache/gravitino/credential/config/TestCOSCredentialConfig.java
b/catalogs/catalog-common/src/test/java/org/apache/gravitino/credential/config/TestCOSCredentialConfig.java
new file mode 100644
index 0000000000..2c01cc2edc
--- /dev/null
+++
b/catalogs/catalog-common/src/test/java/org/apache/gravitino/credential/config/TestCOSCredentialConfig.java
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.credential.config;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.gravitino.credential.CredentialConstants;
+import org.apache.gravitino.storage.COSProperties;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class TestCOSCredentialConfig {
+
+ /**
+ * Baseline properties for a COS STS credential config; individual tests
override the field under
+ * test. Kept minimal (all required-for-STS fields present) so blank-value
tests exercise only the
+ * specific field they target.
+ */
+ private static Map<String, String> baseProps() {
+ Map<String, String> props = new HashMap<>();
+ props.put(COSProperties.GRAVITINO_COS_REGION, "ap-shanghai");
+ props.put(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID, "ak");
+ props.put(COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET, "sk");
+ props.put(COSProperties.GRAVITINO_COS_ROLE_ARN,
"qcs::cam::uin/100:roleName/role");
+ props.put(COSProperties.GRAVITINO_COS_APP_ID, "1259000000");
+ return props;
+ }
+
+ @Test
+ void testTokenExpireDefaultsTo3600WhenAbsent() {
+ // Sanity check: when the caller does not set cos-token-expire-in-secs at
all, the default
+ // (3600s) is applied and the range check does not reject the default.
+ COSCredentialConfig config = new COSCredentialConfig(baseProps());
+ Assertions.assertEquals(3600, config.tokenExpireInSecs());
+ }
+
+ @Test
+ void testTokenExpireAcceptsBoundaryValues() {
+ // 1s and 43200s are the extreme values the range check is expected to
allow.
+ // 43200s (12h) is the documented maximum for Tencent Cloud CAM AssumeRole
DurationSeconds
+ // per tencentcloud-sdk-java-sts v3.1.1239.
+ for (int valid : new int[] {1, 3600, 43200}) {
+ Map<String, String> props =
+ ImmutableMap.<String, String>builder()
+ .putAll(baseProps())
+ .put(CredentialConstants.COS_TOKEN_EXPIRE_IN_SECS,
String.valueOf(valid))
+ .build();
+ COSCredentialConfig config = new COSCredentialConfig(props);
+ Assertions.assertEquals(valid, config.tokenExpireInSecs(), "value=" +
valid);
+ }
+ }
+
+ @Test
+ void testTokenExpireRejectsNonPositiveValues() {
+ // Zero and negative values are meaningless for an STS token TTL. Fail
fast at read time
+ // rather than surfacing a Tencent Cloud STS API error at runtime.
+ //
+ // Note: Config#loadFromMap only records raw strings; ConfigEntry
validators fire lazily on
+ // Config#get. That's why the assertion wraps tokenExpireInSecs() rather
than the constructor.
+ for (int invalid : new int[] {0, -1, -3600}) {
+ Map<String, String> props =
+ ImmutableMap.<String, String>builder()
+ .putAll(baseProps())
+ .put(CredentialConstants.COS_TOKEN_EXPIRE_IN_SECS,
String.valueOf(invalid))
+ .build();
+ COSCredentialConfig config = new COSCredentialConfig(props);
+ Assertions.assertThrows(
+ IllegalArgumentException.class, config::tokenExpireInSecs, "value="
+ invalid);
+ }
+ }
+
+ @Test
+ void testTokenExpireRejectsValuesAboveTencentCloudLimit() {
+ // 43201s and above are guaranteed to be rejected by Tencent Cloud STS at
call time; catching
+ // them here gives a clearer error message tied to the SDK version we
depend on.
+ for (int invalid : new int[] {43201, 86400, Integer.MAX_VALUE}) {
+ Map<String, String> props =
+ ImmutableMap.<String, String>builder()
+ .putAll(baseProps())
+ .put(CredentialConstants.COS_TOKEN_EXPIRE_IN_SECS,
String.valueOf(invalid))
+ .build();
+ COSCredentialConfig config = new COSCredentialConfig(props);
+ Assertions.assertThrows(
+ IllegalArgumentException.class, config::tokenExpireInSecs, "value="
+ invalid);
+ }
+ }
+
+ @Test
+ void testAccessorsReadBaselineProperties() {
+ // Sanity check that the getters return exactly what was fed in via
loadFromMap. Guards
+ // against future refactors accidentally aliasing property keys.
+ COSCredentialConfig config = new COSCredentialConfig(baseProps());
+ Assertions.assertEquals("ap-shanghai", config.region());
+ Assertions.assertEquals("ak", config.accessKeyID());
+ Assertions.assertEquals("sk", config.secretAccessKey());
+ Assertions.assertEquals("qcs::cam::uin/100:roleName/role",
config.cosRoleArn());
+ Assertions.assertEquals("1259000000", config.appID());
+ }
+
+ @Test
+ void testExternalIdIsOptional() {
+ // cos-external-id has no NotBlank check because most tenants do not use
cross-account
+ // AssumeRole. When omitted, externalID() must return null so
COSTokenGenerator can skip
+ // setting ExternalId on the AssumeRole request.
+ COSCredentialConfig config = new COSCredentialConfig(baseProps());
+ Assertions.assertNull(config.externalID());
+
+ Map<String, String> withExtId =
+ ImmutableMap.<String, String>builder()
+ .putAll(baseProps())
+ .put(COSProperties.GRAVITINO_COS_EXTERNAL_ID, "ext-42")
+ .build();
+ COSCredentialConfig withExt = new COSCredentialConfig(withExtId);
+ Assertions.assertEquals("ext-42", withExt.externalID());
+ }
+
+ @Test
+ void testRegionIsRequired() {
+ // A blank / missing cos-region weakens the STS session policy (region ARN
would degrade to
+ // a wildcard). ConfigEntry.checkValue should reject it.
+ for (String blank : new String[] {"", " "}) {
+ Map<String, String> props = new HashMap<>(baseProps());
+ props.put(COSProperties.GRAVITINO_COS_REGION, blank);
+ COSCredentialConfig config = new COSCredentialConfig(props);
+ Assertions.assertThrows(
+ IllegalArgumentException.class, config::region, "region=[" + blank +
"]");
+ }
+ // Missing altogether: reading a required entry with no default must throw
as well.
+ Map<String, String> missing = new HashMap<>(baseProps());
+ missing.remove(COSProperties.GRAVITINO_COS_REGION);
+ COSCredentialConfig config = new COSCredentialConfig(missing);
+ Assertions.assertThrows(RuntimeException.class, config::region);
+ }
+
+ @Test
+ void testRoleArnIsRequiredForStsPath() {
+ // cos-role-arn is what tells the server which CAM role to AssumeRole
into. Without it the
+ // STS provider would call AssumeRole with a null RoleArn and the API
would 4xx. Catching it
+ // in config validation gives a clearer error tied to the property name.
+ for (String blank : new String[] {"", " "}) {
+ Map<String, String> props = new HashMap<>(baseProps());
+ props.put(COSProperties.GRAVITINO_COS_ROLE_ARN, blank);
+ COSCredentialConfig config = new COSCredentialConfig(props);
+ Assertions.assertThrows(
+ IllegalArgumentException.class, config::cosRoleArn, "role-arn=[" +
blank + "]");
+ }
+ }
+
+ @Test
+ void testAppIdIsRequiredForStsPath() {
+ // cos-app-id is required to build the resource ARN "<bucket>-<APPID>"
that scopes the STS
+ // session policy. Without it the policy would allow the wrong bucket, so
we hard-fail early.
+ for (String blank : new String[] {"", " "}) {
+ Map<String, String> props = new HashMap<>(baseProps());
+ props.put(COSProperties.GRAVITINO_COS_APP_ID, blank);
+ COSCredentialConfig config = new COSCredentialConfig(props);
+ Assertions.assertThrows(
+ IllegalArgumentException.class, config::appID, "app-id=[" + blank +
"]");
+ }
+ }
+}
diff --git a/clients/filesystem-hadoop3-runtime/build.gradle.kts
b/clients/filesystem-hadoop3-runtime/build.gradle.kts
index 28aa61be43..fb252ac43d 100644
--- a/clients/filesystem-hadoop3-runtime/build.gradle.kts
+++ b/clients/filesystem-hadoop3-runtime/build.gradle.kts
@@ -43,6 +43,12 @@ tasks.withType<ShadowJar>(ShadowJar::class.java) {
configurations = listOf(project.configurations.runtimeClasspath.get())
archiveClassifier.set("")
+ // Strip shaded slf4j-api brought in by :clients:client-java-runtime —
Hadoop classpaths
+ // usually ship slf4j 1.7.x, and bundling 2.x here breaks that binding at
runtime.
+ exclude("org/slf4j/**")
+ exclude("META-INF/maven/org.slf4j/**")
+ exclude("META-INF/services/org.slf4j.spi.SLF4JServiceProvider")
+
// Relocate dependencies to avoid conflicts
relocate("com.google", "org.apache.gravitino.shaded.com.google") {
// Do not relocate com.google.cloud.hadoop classes — they come from the
external
diff --git
a/clients/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/integration/test/GravitinoVirtualFileSystemCOSCredentialIT.java
b/clients/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/integration/test/GravitinoVirtualFileSystemCOSCredentialIT.java
new file mode 100644
index 0000000000..f2a1b7a818
--- /dev/null
+++
b/clients/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/integration/test/GravitinoVirtualFileSystemCOSCredentialIT.java
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.gravitino.filesystem.hadoop.integration.test;
+
+import static
org.apache.gravitino.catalog.fileset.FilesetCatalogPropertiesMetadata.FILESYSTEM_PROVIDERS;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import org.apache.gravitino.Catalog;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.catalog.hadoop.fs.FileSystemUtils;
+import org.apache.gravitino.cos.fs.COSFileSystemProvider;
+import org.apache.gravitino.credential.COSTokenCredential;
+import org.apache.gravitino.credential.Credential;
+import org.apache.gravitino.credential.CredentialConstants;
+import org.apache.gravitino.file.Fileset;
+import org.apache.gravitino.integration.test.util.GravitinoITUtils;
+import org.apache.gravitino.storage.COSProperties;
+import org.apache.hadoop.conf.Configuration;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIf;
+import org.junit.platform.commons.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Integration test for the GVFS path against a Tencent Cloud COS catalog
whose credentials are
+ * vended dynamically via the {@code cos-token} provider (STS AssumeRole).
Mirrors {@link
+ * GravitinoVirtualFileSystemOSSCredentialIT} so that the two clouds keep
parity in coverage.
+ *
+ * <p>The test is gated on a separate set of {@code *_FOR_CREDENTIAL} env vars
(and notably {@code
+ * COS_ROLE_ARN_FOR_CREDENTIAL}) so that the static-key COS IT and this STS IT
can coexist without
+ * accidentally running with stale env config. If {@code
COS_EXTERNAL_ID_FOR_CREDENTIAL} is also
+ * set, it is propagated to the catalog to exercise the optional {@code
cos-external-id} path.
+ */
+@EnabledIf(value = "cosIsConfigured", disabledReason = "Tencent Cloud COS STS
is not prepared")
+public class GravitinoVirtualFileSystemCOSCredentialIT extends
GravitinoVirtualFileSystemIT {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(GravitinoVirtualFileSystemCOSCredentialIT.class);
+
+ public static final String BUCKET_NAME =
System.getenv("COS_BUCKET_NAME_FOR_CREDENTIAL");
+ public static final String COS_ACCESS_KEY =
System.getenv("COS_ACCESS_KEY_ID_FOR_CREDENTIAL");
+ public static final String COS_SECRET_KEY =
System.getenv("COS_SECRET_ACCESS_KEY_FOR_CREDENTIAL");
+ public static final String COS_REGION =
System.getenv("COS_REGION_FOR_CREDENTIAL");
+ public static final String COS_ENDPOINT =
System.getenv("COS_ENDPOINT_FOR_CREDENTIAL");
+ public static final String COS_ROLE_ARN =
System.getenv("COS_ROLE_ARN_FOR_CREDENTIAL");
+ // COSCredentialConfig#COS_APP_ID has a NotBlank check and COSTokenGenerator
uses it to build
+ // the STS resource ARN (e.g. "my-bucket-1259000000"). Missing it would fail
catalog creation
+ // before any GVFS traffic runs.
+ public static final String COS_APP_ID =
System.getenv("COS_APP_ID_FOR_CREDENTIAL");
+ // Optional external id for AssumeRole; when set the catalog is created with
it so that STS
+ // rejects calls whose ExternalId does not match the target role's trust
policy.
+ public static final String COS_EXTERNAL_ID =
System.getenv("COS_EXTERNAL_ID_FOR_CREDENTIAL");
+
+ @BeforeAll
+ public void startIntegrationTest() {
+ // Override parent's @BeforeAll - it's redirected to startUp() below so
that we can copy the
+ // tencent-bundle JARs to the Gravitino server before booting it.
+ }
+
+ @BeforeAll
+ public void startUp() throws Exception {
+ copyBundleJarsToHadoop("tencent-bundle");
+ super.startIntegrationTest();
+
+ // hadoop-cos defaults to 128 MB blocks; must match
GravitinoVirtualFileSystemCOSIT so that
+ // the parent's testGetDefaultBlockSizes() assertion passes.
+ defaultBlockSize = 128 * 1024 * 1024;
+ defaultReplication = 1;
+
+ metalakeName = GravitinoITUtils.genRandomName("gvfs_it_metalake");
+ catalogName = GravitinoITUtils.genRandomName("catalog");
+ schemaName = GravitinoITUtils.genRandomName("schema");
+
+ Assertions.assertFalse(client.metalakeExists(metalakeName));
+ metalake = client.createMetalake(metalakeName, "metalake comment",
Collections.emptyMap());
+ Assertions.assertTrue(client.metalakeExists(metalakeName));
+
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(FILESYSTEM_PROVIDERS, "cos");
+ properties.put(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID, COS_ACCESS_KEY);
+ properties.put(COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET,
COS_SECRET_KEY);
+ properties.put(COSProperties.GRAVITINO_COS_REGION, COS_REGION);
+ if (StringUtils.isNotBlank(COS_ENDPOINT)) {
+ properties.put(COSProperties.GRAVITINO_COS_ENDPOINT, COS_ENDPOINT);
+ }
+ properties.put(COSProperties.GRAVITINO_COS_ROLE_ARN, COS_ROLE_ARN);
+ properties.put(COSProperties.GRAVITINO_COS_APP_ID, COS_APP_ID);
+ if (StringUtils.isNotBlank(COS_EXTERNAL_ID)) {
+ properties.put(COSProperties.GRAVITINO_COS_EXTERNAL_ID, COS_EXTERNAL_ID);
+ }
+ properties.put(
+ CredentialConstants.CREDENTIAL_PROVIDERS,
COSTokenCredential.COS_TOKEN_CREDENTIAL_TYPE);
+ // Explicit non-default TTL so downstream tests can assert expireTimeInMs
propagation without
+ // colliding with the 3600s built-in default.
+ properties.put(CredentialConstants.COS_TOKEN_EXPIRE_IN_SECS, "1800");
+
+ Catalog catalog =
+ metalake.createCatalog(
+ catalogName, Catalog.Type.FILESET, "hadoop", "catalog comment",
properties);
+ Assertions.assertTrue(metalake.catalogExists(catalogName));
+
+ catalog.asSchemas().createSchema(schemaName, "schema comment", properties);
+ Assertions.assertTrue(catalog.asSchemas().schemaExists(schemaName));
+
+ conf.set("fs.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem");
+ conf.set("fs.AbstractFileSystem.gvfs.impl",
"org.apache.gravitino.filesystem.hadoop.Gvfs");
+ conf.set("fs.gvfs.impl.disable.cache", "true");
+ conf.set("fs.gravitino.server.uri", serverUri);
+ conf.set("fs.gravitino.client.metalake", metalakeName);
+ // Enable credential vending on the GVFS client so that data-plane
operations go through the
+ // cos-token STS AssumeRole flow (via COSTokenGenerator) instead of
falling back to the static
+ // AK/SK configured on the catalog. Mirrors the S3/OSS/ABS/GCS credential
ITs.
+ conf.set("fs.gravitino.enableCredentialVending", "true");
+
+ // Pass the COS endpoint settings to the underlying CosFileSystem so that
GVFS can hand off
+ // file IO once the STS credentials are vended back to the client.
+ conf.set(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID, COS_ACCESS_KEY);
+ conf.set(COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET, COS_SECRET_KEY);
+ conf.set(COSProperties.GRAVITINO_COS_REGION, COS_REGION);
+ if (StringUtils.isNotBlank(COS_ENDPOINT)) {
+ conf.set(COSProperties.GRAVITINO_COS_ENDPOINT, COS_ENDPOINT);
+ }
+ conf.set("fs.cosn.impl", "org.apache.hadoop.fs.CosFileSystem");
+ }
+
+ @AfterAll
+ public void tearDown() throws IOException {
+ Catalog catalog = metalake.loadCatalog(catalogName);
+ catalog.asSchemas().dropSchema(schemaName, true);
+ metalake.dropCatalog(catalogName, true);
+ client.dropMetalake(metalakeName, true);
+
+ if (client != null) {
+ client.close();
+ client = null;
+ }
+
+ try {
+ closer.close();
+ } catch (Exception e) {
+ LOG.error("Exception in closing CloseableGroup", e);
+ }
+ }
+
+ /**
+ * Strip the {@code gravitino.bypass} prefix and translate Gravitino
property keys into their
+ * hadoop-cos equivalents. Mirrors the OSS counterpart and the production
{@code
+ * GravitinoVirtualFileSystem#getConfigMap}.
+ */
+ protected Configuration
convertGvfsConfigToRealFileSystemConfig(Configuration gvfsConf) {
+ Configuration cosConf = new Configuration();
+ Map<String, String> map = Maps.newHashMap();
+
+ gvfsConf.forEach(entry -> map.put(entry.getKey(), entry.getValue()));
+
+ Map<String, String> hadoopConfMap =
+ FileSystemUtils.toHadoopConfigMap(
+ map, COSFileSystemProvider.GRAVITINO_KEY_TO_COS_HADOOP_KEY);
+
+ hadoopConfMap.forEach(cosConf::set);
+
+ return cosConf;
+ }
+
+ protected String genStorageLocation(String fileset) {
+ return String.format("cosn://%s/%s", BUCKET_NAME, fileset);
+ }
+
+ @Disabled(
+ "COS does not support HDFS-style append; CosFileSystem throws "
+ + "UnsupportedOperationException for append()")
+ public void testAppend() throws IOException {}
+
+ /**
+ * For the {@code cos-token} STS provider, catalog-scoped credential vending
must return an empty
+ * array (mirrors {@code oss-token} / {@code s3-token}).
+ *
+ * <p>Rationale: {@code COSTokenGenerator.generate(...)} bails out with
{@code null} whenever the
+ * context is not a {@code PathBasedCredentialContext}, and the
catalog-level endpoint always
+ * hands in a {@code CatalogCredentialContext}. Without this test we would
silently regress if
+ * someone later flipped that guard - the STS token would leak out at
catalog scope with a policy
+ * that is not path-restricted.
+ */
+ @Test
+ void testCatalogCredentialsReturnsEmpty() {
+ Catalog catalog = metalake.loadCatalog(catalogName);
+ Credential[] credentials = catalog.supportsCredentials().getCredentials();
+ Assertions.assertEquals(
+ 0,
+ credentials.length,
+ "cos-token provider must return no credential at catalog scope
(path-less context)");
+ }
+
+ /**
+ * {@code cos-token-expire-in-secs} configured on the catalog (1800 in
{@link #startUp()}) must be
+ * honoured end-to-end - i.e. the vended {@code expireTimeInMs} sits ~30
minutes in the future
+ * rather than falling back to the 3600s built-in default. Also asserts the
three cos-token info
+ * fields are present so downstream consumers can rely on the payload shape.
+ */
+ @Test
+ void testFilesetCredentialRespectsExpireInSecs() {
+ String filesetName = GravitinoITUtils.genRandomName("cos_cred_ttl");
+ NameIdentifier filesetIdent = NameIdentifier.of(schemaName, filesetName);
+ Catalog catalog = metalake.loadCatalog(catalogName);
+
+ Fileset fileset =
+ catalog
+ .asFilesetCatalog()
+ .createFileset(
+ filesetIdent,
+ "fileset for ttl check",
+ Fileset.Type.MANAGED,
+ genStorageLocation(filesetName),
+ ImmutableMap.of());
+
+ try {
+ long now = System.currentTimeMillis();
+ Credential[] credentials =
fileset.supportsCredentials().getCredentials();
+
+ Assertions.assertEquals(1, credentials.length, "expect exactly one
cos-token credential");
+ Assertions.assertInstanceOf(COSTokenCredential.class, credentials[0]);
+
+ Map<String, String> info = credentials[0].credentialInfo();
+ Assertions.assertTrue(
+
StringUtils.isNotBlank(info.get(COSTokenCredential.GRAVITINO_COS_SESSION_ACCESS_KEY_ID)),
+ "cos-access-key-id must be present");
+ Assertions.assertTrue(
+ StringUtils.isNotBlank(
+
info.get(COSTokenCredential.GRAVITINO_COS_SESSION_SECRET_ACCESS_KEY)),
+ "cos-secret-access-key must be present");
+ Assertions.assertTrue(
+
StringUtils.isNotBlank(info.get(COSTokenCredential.GRAVITINO_COS_SESSION_TOKEN)),
+ "cos-security-token must be present");
+
+ long remainingMs = credentials[0].expireTimeInMs() - now;
+ // We configured 1800s. Allow a generous +/- 120s window to absorb the
round trip to STS and
+ // any small clock skew between server and this JVM.
+ long lower = 1_680_000L;
+ long upper = 1_920_000L;
+ Assertions.assertTrue(
+ remainingMs >= lower && remainingMs <= upper,
+ () ->
+ String.format(
+ "expireTimeInMs remaining=%d ms is outside [%d, %d] -
cos-token-expire-in-secs=1800 did not propagate",
+ remainingMs, lower, upper));
+ } finally {
+ catalog.asFilesetCatalog().dropFileset(filesetIdent);
+ }
+ }
+
+ protected static boolean cosIsConfigured() {
+ return
StringUtils.isNotBlank(System.getenv("COS_ACCESS_KEY_ID_FOR_CREDENTIAL"))
+ &&
StringUtils.isNotBlank(System.getenv("COS_SECRET_ACCESS_KEY_FOR_CREDENTIAL"))
+ &&
StringUtils.isNotBlank(System.getenv("COS_BUCKET_NAME_FOR_CREDENTIAL"))
+ && StringUtils.isNotBlank(System.getenv("COS_REGION_FOR_CREDENTIAL"))
+ && StringUtils.isNotBlank(System.getenv("COS_ROLE_ARN_FOR_CREDENTIAL"))
+ && StringUtils.isNotBlank(System.getenv("COS_APP_ID_FOR_CREDENTIAL"));
+ }
+}
diff --git
a/common/src/test/java/org/apache/gravitino/credential/TestCredentialFactory.java
b/common/src/test/java/org/apache/gravitino/credential/TestCredentialFactory.java
index eaa7402c2a..4feb4b4e49 100644
---
a/common/src/test/java/org/apache/gravitino/credential/TestCredentialFactory.java
+++
b/common/src/test/java/org/apache/gravitino/credential/TestCredentialFactory.java
@@ -163,6 +163,54 @@ public class TestCredentialFactory {
Assertions.assertEquals(expireTime,
ossSecretKeyCredential1.expireTimeInMs());
}
+ @Test
+ void testCOSSecretKeyCredential() {
+ Map<String, String> cosSecretKeyCredentialInfo =
+ ImmutableMap.of(
+ COSSecretKeyCredential.GRAVITINO_COS_STATIC_ACCESS_KEY_ID,
+ "accessKeyId",
+ COSSecretKeyCredential.GRAVITINO_COS_STATIC_SECRET_ACCESS_KEY,
+ "secretAccessKey");
+ long expireTime = 0;
+ Credential cosSecretKeyCredential =
+ CredentialFactory.create(
+ COSSecretKeyCredential.COS_SECRET_KEY_CREDENTIAL_TYPE,
+ cosSecretKeyCredentialInfo,
+ expireTime);
+ Assertions.assertEquals(
+ COSSecretKeyCredential.COS_SECRET_KEY_CREDENTIAL_TYPE,
+ cosSecretKeyCredential.credentialType());
+ Assertions.assertInstanceOf(COSSecretKeyCredential.class,
cosSecretKeyCredential);
+ COSSecretKeyCredential typed = (COSSecretKeyCredential)
cosSecretKeyCredential;
+ Assertions.assertEquals("accessKeyId", typed.accessKeyId());
+ Assertions.assertEquals("secretAccessKey", typed.secretAccessKey());
+ Assertions.assertEquals(expireTime, typed.expireTimeInMs());
+ }
+
+ @Test
+ void testCOSTokenCredential() {
+ Map<String, String> cosTokenCredentialInfo =
+ ImmutableMap.of(
+ COSTokenCredential.GRAVITINO_COS_SESSION_ACCESS_KEY_ID,
+ "access-id",
+ COSTokenCredential.GRAVITINO_COS_SESSION_SECRET_ACCESS_KEY,
+ "secret-key",
+ COSTokenCredential.GRAVITINO_COS_SESSION_TOKEN,
+ "token");
+ long expireTime = 100;
+ Credential cosTokenCredential =
+ CredentialFactory.create(
+ COSTokenCredential.COS_TOKEN_CREDENTIAL_TYPE,
cosTokenCredentialInfo, expireTime);
+ Assertions.assertEquals(
+ COSTokenCredential.COS_TOKEN_CREDENTIAL_TYPE,
cosTokenCredential.credentialType());
+ Assertions.assertInstanceOf(COSTokenCredential.class, cosTokenCredential);
+ COSTokenCredential typed = (COSTokenCredential) cosTokenCredential;
+ Assertions.assertEquals("access-id", typed.accessKeyId());
+ Assertions.assertEquals("secret-key", typed.secretAccessKey());
+ Assertions.assertEquals("token", typed.securityToken());
+ Assertions.assertEquals(expireTime, typed.expireTimeInMs());
+ }
+
@Test
void testADLSTokenCredential() {
String storageAccountName = "storage-account-name";
diff --git a/docs/fileset-catalog-with-cos.md b/docs/fileset-catalog-with-cos.md
index dfe533d173..cff8e3e9cf 100644
--- a/docs/fileset-catalog-with-cos.md
+++ b/docs/fileset-catalog-with-cos.md
@@ -39,13 +39,17 @@ These properties are needed in addition to the shared
the GVFS clients, so they are listed together here — note that the Python
client spells them with
underscores while the catalog and the Java client use hyphens.
-| Catalog and Java client | Python client | Description
| Required |
-|-------------------------|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|
-| `cos-region` | `cos_region` | Region of the COS
bucket, for example `ap-guangzhou` or `ap-shanghai`.
| Yes |
-| `cos-endpoint` | `cos_endpoint` | Endpoint *suffix* of the
COS service, mapped to `fs.cosn.bucket.endpoint_suffix`. It is a host suffix,
not a URL — `cos.ap-guangzhou.myqcloud.com`, not
`https://cos.ap-guangzhou.myqcloud.com`. When unset, hadoop-cos derives it from
`cos-region`. Set it only to reach a non-public endpoint such as a VPC
endpoint. | No |
-| `cos-access-key-id` | `cos_access_key_id` | Static access key id,
the Tencent Cloud `SecretId`.
| Yes |
-| `cos-secret-access-key` | `cos_secret_access_key` | Static secret access
key, the Tencent Cloud `SecretKey`.
| Yes |
-| `credential-providers` | (n/a) | The credential provider
types, separated by comma. Possible values are `cos-secret-key`. Setting it
enables credential vending, so clients no longer need the credentials above.
See [credential vending](./security/credential-vending.md) for the extra
properties each provider takes. | No |
+| Catalog and Java client | Python client | Description
| Required |
+|----------------------------|----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|
+| `cos-region` | `cos_region` | Region of the COS
bucket, for example `ap-guangzhou` or `ap-shanghai`.
| Yes |
+| `cos-endpoint` | `cos_endpoint` | Endpoint *suffix*
of the COS service, mapped to `fs.cosn.bucket.endpoint_suffix`. It is a host
suffix, not a URL — `cos.ap-guangzhou.myqcloud.com`, not
`https://cos.ap-guangzhou.myqcloud.com`. When unset, hadoop-cos derives it from
`cos-region`. Set it only to reach a non-public endpoint such as a VPC
endpoint.
| No |
+| `cos-access-key-id` | `cos_access_key_id` | Static access key
id, the Tencent Cloud `SecretId`.
| Yes |
+| `cos-secret-access-key` | `cos_secret_access_key` | Static secret
access key, the Tencent Cloud `SecretKey`.
| Yes |
+| `credential-providers` | (n/a) | The credential
provider types, separated by comma. Supported values are `cos-secret-key`
(static AK/SK vended by the server) and `cos-token` (short-lived STS token
issued via CAM `AssumeRole`). Setting it enables credential vending, so clients
no longer need the credentials above. See [credential
vending](./security/credential-vending.md) for the extra properties each
provider takes.| No |
+| `cos-role-arn` | `cos_role_arn` | The CAM role ARN
that the Gravitino server assumes when issuing STS temporary credentials, e.g.
`qcs::cam::uin/100012345678:roleName/GravitinoCOSAccess`. Required only when
`credential-providers` includes `cos-token`.
| No |
+| `cos-app-id` | `cos_app_id` | The numeric
Tencent Cloud AppId of the bucket owner (the trailing segment of the bucket
name, e.g. `1250000000`). Required only when `credential-providers` includes
`cos-token`; used to build the resource ARN in the STS session policy.
| No |
+| `cos-external-id` | `cos_external_id` | Optional
`ExternalId` propagated to the STS `AssumeRole` call to lock the role's trust
policy to Gravitino. Only meaningful when `credential-providers` includes
`cos-token`.
| No |
+| `cos-token-expire-in-secs` | `cos_token_expire_in_secs` | The COS STS token
expire time in seconds. Must not exceed the role's max session duration. Only
meaningful when `credential-providers` includes `cos-token`. Defaults to
`3600`.
| No |
:::note
`default-filesystem-provider` and `filesystem-providers` are deprecated. The
fileset catalog
@@ -451,17 +455,16 @@ For further Java client use cases, see
## Credential Vending
-With credential vending the catalog holds the Tencent Cloud COS credentials
and the Gravitino server hands
-out a credential per request, so clients never hold cloud keys of their own.
See
+With credential vending the catalog holds the Tencent Cloud COS credentials
and the Gravitino
+server hands out a credential per request, so clients never hold cloud keys of
their own. See
[Credential Vending](./security/credential-vending.md) for the general
mechanism.
-The supported provider is `cos-secret-key`, which vends the static
-`cos-access-key-id` / `cos-secret-access-key` configured on the catalog.
+The currently supported credential providers are listed below:
-:::note
-STS-token-based vending (`cos-token`), including role-arn and assume-role
configuration, will be
-added in a follow-up release. Until then only `cos-secret-key` is available.
-:::
+| Credential provider | Description
| Vended credential type |
+|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
+| `cos-secret-key` | The Gravitino server hands out the static
`cos-access-key-id` / `cos-secret-access-key` configured on the catalog. Useful
for centralising credentials on the server side.
| Static AK/SK |
+| `cos-token` | The Gravitino server calls Tencent Cloud CAM
`AssumeRole` and hands out a short-lived STS triple (`TmpSecretId` /
`TmpSecretKey` / `SessionToken`), scoped down to the fileset paths that the
client requested. Requires `cos-role-arn` and `cos-app-id` on the catalog and
CAM permissions (`sts:AssumeRole` / `cam:GetFederationToken`) on the account
whose AK/SK is configured on the catalog. | Short-lived STS token |
### Configure the catalog, schema, and fileset
@@ -482,6 +485,31 @@ curl -X POST -H "Accept:
application/vnd.gravitino.v1+json" \
}' http://localhost:8090/api/metalakes/metalake/catalogs
```
+To enable STS-based credential vending instead (recommended for production,
since the AK/SK on the catalog never leaves the server), switch
`credential-providers` to `cos-token` and add the CAM role wiring. The AK/SK
below is the *server-side* account used to call `sts:AssumeRole`; the vended
clients will only ever see the short-lived session token issued for the
requested fileset paths:
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+-H "Content-Type: application/json" -d '{
+ "name": "cos-catalog-with-sts-vending",
+ "type": "FILESET",
+ "comment": "This is a COS fileset catalog with STS credential vending",
+ "properties": {
+ "location": "cosn://my-bucket-1250000000/root",
+ "cos-region": "ap-guangzhou",
+ "cos-access-key-id": "server_access_key",
+ "cos-secret-access-key": "server_secret_key",
+ "credential-providers": "cos-token",
+ "cos-role-arn": "qcs::cam::uin/100012345678:roleName/GravitinoCOSAccess",
+ "cos-app-id": "1250000000",
+ "cos-token-expire-in-secs": "1800"
+ }
+}' http://localhost:8090/api/metalakes/metalake/catalogs
+```
+
+:::note
+When using `cos-token`, the CAM role referenced by `cos-role-arn` must (1)
trust the Gravitino server principal (or use `cos-external-id` for stricter
matching) and (2) have COS read/write permissions on the bucket paths you plan
to expose. Gravitino narrows the vended token further via a session policy that
only allows the fileset's own read/write locations, so the role can be as broad
as your organisation's policies allow — the effective permission is the
intersection.
+:::
+
Create the schema and fileset in the credential-vending catalog:
```shell
diff --git a/docs/security/credential-vending.md
b/docs/security/credential-vending.md
index ea8f11306b..c3283678e7 100755
--- a/docs/security/credential-vending.md
+++ b/docs/security/credential-vending.md
@@ -13,14 +13,14 @@ Gravitino credential vending is used to generate temporary
or static credentials
| Catalog type | Vends |
|--------------|---------------------------------|
-| Fileset | S3, OSS, GCS, ADLS |
+| Fileset | S3, OSS, GCS, ADLS, COS |
| Hive | S3, OSS, GCS, ADLS |
| Iceberg | S3, OSS, GCS, ADLS |
| Glue | S3 |
| JDBC | JDBC user and password |
| Paimon | S3, OSS, JDBC user and password |
-S3 is Amazon S3, OSS is Alibaba Cloud OSS, GCS is Google Cloud Storage, and
ADLS is Azure Data Lake Storage. The Gravitino Spark, Flink, and Trino
connectors consume vended credentials automatically for these catalogs.
+S3 is Amazon S3, OSS is Alibaba Cloud OSS, GCS is Google Cloud Storage, ADLS
is Azure Data Lake Storage, and COS is Tencent Cloud COS. The Gravitino Spark,
Flink, and Trino connectors consume vended credentials automatically for these
catalogs.
## Quick Start
@@ -113,6 +113,8 @@ Catalogs defined in `gravitino.conf` are not registered in
a metalake, so Gravit
| `adls-token` | ADLS | A user delegation SAS token
|
| `azure-account-key` | ADLS | The configured static storage account key
|
| `gcs-token` | GCS | A downscoped access token
|
+| `cos-token` | COS | A temporary STS token
|
+| `cos-secret-key` | COS | The configured static access key and secret
|
| `jdbc-user-password` | JDBC | The configured JDBC username and password
|
Each value has its own properties, listed in the sections below. To vend for
more than one storage type on a catalog, separate values with a comma. Custom
providers can be added by implementing `CredentialProvider`, described under
[Custom Credentials](#custom-credentials).
@@ -278,6 +280,85 @@ The key is long-lived, carries whatever permissions its
RAM user has, and is not
| `oss-access-key-id` | The static access key ID used to access OSS data.
| (none) | Yes |
| `oss-secret-access-key` | The static secret access key used to access OSS
data. | (none) | Yes |
+## COS
+
+### `cos-token`
+
+Gravitino calls Tencent Cloud STS
[AssumeRole](https://www.tencentcloud.com/document/product/598/33416) and
returns temporary credentials scoped to the table path. The role is a Cloud
Access Management (CAM) role on Tencent Cloud; the CAM abbreviation is used
throughout this section.
+
+Also set `cos-access-key-id` and `cos-secret-access-key`. Gravitino uses them
to call AssumeRole, not to reach data, and they are never sent to the engine.
+
+| Property | Description
|
Default value | Required |
+|----------------------------|-----------------------------------------------------------------------------------------------------------------------------|---------------|----------|
+| `cos-access-key-id` | The static access key ID (Tencent Cloud
`SecretId`) used by Gravitino to call STS `AssumeRole`.
| (none) | Yes |
+| `cos-secret-access-key` | The static secret access key (Tencent Cloud
`SecretKey`) used by Gravitino to call STS `AssumeRole`.
| (none) | Yes |
+| `cos-role-arn` | The ARN of the CAM role to assume, e.g.
`qcs::cam::uin/100012345678:roleName/GravitinoCOSAccess`.
| (none) | Yes |
+| `cos-region` | The region of the bucket, e.g. `ap-guangzhou`.
Used to build the STS endpoint and the resource ARN. |
(none) | Yes |
+| `cos-app-id` | The numeric Tencent Cloud AppId of the bucket
owner (the trailing segment of the bucket name, e.g. `1250000000`). |
(none) | Yes |
+| `cos-external-id` | Optional `ExternalId` propagated to STS
`AssumeRole` to lock the role's trust policy to Gravitino.
| (none) | No |
+| `cos-token-expire-in-secs` | The COS security token expire time in secs.
Must not exceed the role's max session duration.
| 3600 | No |
+
+#### Trust Policy on the CAM Role
+
+The role in `cos-role-arn` must allow the `cos-access-key-id` principal to
assume it. If `cos-external-id` is set, the trust policy must require the same
value.
+
+```json
+{
+ "version": "2.0",
+ "statement": [{
+ "effect": "allow",
+ "action": "name/sts:AssumeRole",
+ "principal": { "qcs": ["qcs::cam::uin/{account_uin}:uin/{account_uin}"] },
+ "condition": {
+ "string_equal": { "sts:external_id": "{external_id}" }
+ }
+ }]
+}
+```
+
+#### Permission Policy on the CAM Role
+
+The vended credentials inherit this policy, narrowed to the table path.
+
+```json
+{
+ "version": "2.0",
+ "statement": [
+ {
+ "effect": "allow",
+ "action": [
+ "cos:GetObject",
+ "cos:HeadObject",
+ "cos:PutObject",
+ "cos:DeleteObject",
+ "cos:InitiateMultipartUpload",
+ "cos:UploadPart",
+ "cos:ListParts",
+ "cos:CompleteMultipartUpload",
+ "cos:AbortMultipartUpload"
+ ],
+ "resource":
"qcs::cos:{region}:uid/{app_id}:{bucket_name}-{app_id}/{warehouse_path}/*"
+ },
+ {
+ "effect": "allow",
+ "action": ["cos:GetBucket", "cos:HeadBucket", "cos:GetBucketLocation"],
+ "resource": "qcs::cos:{region}:uid/{app_id}:{bucket_name}-{app_id}/*"
+ }
+ ]
+}
+```
+
+### `cos-secret-key`
+
+Returns the catalog's configured access key and secret to the client,
unchanged.
+
+The key is long-lived, carries whatever permissions its CAM user has, and is
not scoped to the table path. Any client that can load a table receives it, and
it stays valid after the query finishes. Prefer `cos-token`. Use
`cos-secret-key` to confirm the vending path works before configuring a role.
+
+| Property | Description
| Default value | Required |
+|-------------------------|-------------------------------------------------------|---------------|----------|
+| `cos-access-key-id` | The static access key ID used to access COS data.
| (none) | Yes |
+| `cos-secret-access-key` | The static secret access key used to access COS
data. | (none) | Yes |
+
## ADLS
### `adls-token`
diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml
index cb83b3b861..6224b289d8 100644
--- a/gradle/libs.versions.toml
+++ b/gradle/libs.versions.toml
@@ -156,6 +156,7 @@ hudi = "0.15.0"
google-auth = "1.28.0"
aliyun-credentials = "0.3.12"
aliyun-sdk-oss = "3.10.2"
+tencentcloud-sdk-sts = "3.1.1239"
openlineage = "1.29.0"
jcstress = "0.8.15"
jmh-plugin = "0.7.3"
@@ -355,6 +356,7 @@ google-auth-credentials = { group = "com.google.auth", name
= "google-auth-libra
aliyun-credentials-sdk = { group='com.aliyun', name='credentials-java',
version.ref='aliyun-credentials' }
aliyun-sdk-oss = { module = "com.aliyun.oss:aliyun-sdk-oss", version.ref =
"aliyun-sdk-oss" }
+tencentcloud-sdk-sts = { group = "com.tencentcloudapi", name =
"tencentcloud-sdk-java-sts", version.ref = "tencentcloud-sdk-sts" }
flinkjdbc = {group='org.apache.flink',name='flink-connector-jdbc',
version.ref='flinkjdbc'}
flinkjdbc18 = {group='org.apache.flink',name='flink-connector-jdbc',
version.ref='flinkjdbc18'}
flinkjdbc119 = {group='org.apache.flink',name='flink-connector-jdbc',
version.ref='flinkjdbc119'}
