This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4ce25c68433 [test](fe) Cover credential-aware FileSystem cache keys
for object stores (#68803)
4ce25c68433 is described below
commit 4ce25c68433b366f84bbeb4d1cd92dedb3d18b7a
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Oct 9 13:04:18 2026 +0800
[test](fe) Cover credential-aware FileSystem cache keys for object stores
(#68803)
### What problem does this PR solve?
Issue Number: None
Related PR: #65586, #66315
Problem Summary:
**In short.** This ports the test coverage of branch-4.1 #65586
(credential-aware Hadoop FileSystem cache) to master. master already has
the feature (#66315), but the assertions that guard the object-store
Hadoop maps and the merge / vended-credential paths did not come along.
Only tests are added; no production code changes.
**Background.** #65586 stopped force-disabling the Hadoop FileSystem
cache (`fs.<scheme>.impl.disable.cache=true`) and instead made the cache
credential-aware: the Doris-patched `FileSystem.Cache.Key` folds a
fingerprint of the storage definition into the key. master ported this
in #66315 with a different layout. Instead of one scheme-less
`doris.fs.cache.key`, every storage publishes its fingerprint under each
scheme it serves (`doris.fs.cache.key.<scheme>`), so a merge of several
storages keeps all of them and no combined fingerprint is needed.
| | branch-4.1 #65586 | master #66315 |
|---|---|---|
| Cache key property | `doris.fs.cache.key` |
`doris.fs.cache.key.<scheme>`, one per scheme the storage serves |
| Several storages in one map | combined fingerprint | entries simply
coexist |
| Blanket `fs.<scheme>.impl.disable.cache=true` | removed | removed |
**The problem, and what it cost.** On master nothing asserts the shape
of the S3-family `toHadoopConfigurationMap()`, which is the map the
Hadoop FileSystem actually reads. The 4.1 `testS3DisableHadoopCache`
assertions (COS, GCS, OBS, OSS, S3) have no master counterpart, and
`StorageAdapterFsCacheFingerprintTest#testNoBlanketDisableCacheByDefault`
inspects the `AWS_*` backend map, which never held these flags. Today
every FE unit test still passes if a provider re-adds a blanket disable
flag (silently turning the cache off again), or publishes its
fingerprint under fewer schemes than it is opened with (COS is addressed
as `cos://` but opened as `s3a`, so a key published only under `cos` is
never read, and two catalogs with different credentials can share a
FileSystem). The merge through `CredentialUtils` and the
vended-credential overlay were likewise only checked with mocks or not
at all.
**How this PR fixes it.** It adds the 4.1 assertions in master's terms:
| Test | New assertion |
|---|---|
| `S3` / `Cos` / `Obs` / `Oss` / `GcsFileSystemPropertiesTest` |
`toHadoopConfigurationMap()` has no `fs.<scheme>.impl.disable.cache`; it
carries `doris.fs.cache.key.<scheme>` equal to `fsCacheFingerprint()`
for every scheme the storage serves (S3 `{s3, s3a, s3n}`, COS `{cos,
cosn, s3, s3a}`, OSS `{oss, s3, s3a}`, OBS `{obs, s3, s3a}`, GCS `{gs,
s3, s3a}`); it has no scheme-less key; different credentials give a
different fingerprint |
| `CredentialUtilsTest` | merging real HDFS and S3 adapters through
`getBackendPropertiesFromStorageMap` keeps `doris.fs.cache.key.hdfs` and
`doris.fs.cache.key.s3a` with each storage's own fingerprint |
| `DefaultConnectorContextVendTest` | a vended OSS token carries its own
`doris.fs.cache.key.oss` / `.s3a`, and a rotated access key yields
different values |
**Results.** The new tests pass on current master and fail if any of the
regressions above is introduced.
Original author: @CalvinKirs
---
.../connector/DefaultConnectorContextVendTest.java | 29 +++++++++++++++++++++
.../credentials/CredentialUtilsTest.java | 30 ++++++++++++++++++++++
.../cos/CosFileSystemPropertiesTest.java | 28 ++++++++++++++++++++
.../gcs/GcsFileSystemPropertiesTest.java | 27 +++++++++++++++++++
.../obs/ObsFileSystemPropertiesTest.java | 28 ++++++++++++++++++++
.../oss/OssFileSystemPropertiesTest.java | 28 ++++++++++++++++++++
.../filesystem/s3/S3FileSystemPropertiesTest.java | 28 ++++++++++++++++++++
7 files changed, 198 insertions(+)
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextVendTest.java
b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextVendTest.java
index b8314046dd5..3a282589205 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextVendTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextVendTest.java
@@ -17,6 +17,8 @@
package org.apache.doris.connector;
+import org.apache.doris.filesystem.properties.FsCacheKeys;
+
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -60,6 +62,33 @@ public class DefaultConnectorContextVendTest {
Assertions.assertEquals("testSessionToken789", be.get("AWS_TOKEN"));
}
+ @Test
+ public void vendedTokenCarriesItsOwnFsCacheKey() {
+ Map<String, String> be =
context().vendStorageCredentials(ossToken("STS.testAccessKey123"));
+ Map<String, String> rotated =
context().vendStorageCredentials(ossToken("STS.rotatedAccessKey"));
+
+ // WHY: a vended token is overlaid on the catalog's static storage
properties, and the patched
+ // Hadoop FileSystem caches by doris.fs.cache.key.<scheme>. The
overlay must therefore carry the
+ // token's own fingerprint, or a FileSystem opened with one token
would be reused for another.
+ // MUTATION: dropping the key from the vended map, or deriving it from
anything but the token
+ // -> missing or equal values -> red.
+ for (String scheme : new String[] {"oss", "s3a"}) {
+ String key = FsCacheKeys.fsCacheKeyProperty(scheme);
+ Assertions.assertNotNull(be.get(key), key);
+ Assertions.assertNotEquals(be.get(key), rotated.get(key), key);
+ }
+ Assertions.assertNull(be.get(FsCacheKeys.FS_CACHE_KEY_PROPERTY));
+ }
+
+ private static Map<String, String> ossToken(String accessKeyId) {
+ Map<String, String> token = new HashMap<>();
+ token.put("fs.oss.accessKeyId", accessKeyId);
+ token.put("fs.oss.accessKeySecret", "testSecretKey456");
+ token.put("fs.oss.securityToken", "testSessionToken789");
+ token.put("fs.oss.endpoint", "oss-cn-beijing.aliyuncs.com");
+ return token;
+ }
+
@Test
public void emptyOrNullInputYieldsEmpty() {
// WHY: a non-REST / no-token table passes an empty map; the bridge
must short-circuit to
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/credentials/CredentialUtilsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/credentials/CredentialUtilsTest.java
index 67856553076..41a8d405d77 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/credentials/CredentialUtilsTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/credentials/CredentialUtilsTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.datasource.credentials;
import org.apache.doris.datasource.storage.StorageAdapter;
import org.apache.doris.datasource.storage.StorageTypeId;
+import org.apache.doris.filesystem.properties.FsCacheKeys;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -196,6 +197,35 @@ public class CredentialUtilsTest {
Assertions.assertEquals("hdfs://namenode:9000",
result.get("HDFS_NAMENODE"));
}
+ @Test
+ public void
testGetBackendPropertiesFromStorageMapKeepsEveryStoragesFsCacheKey() {
+ // Real adapters rather than mocks: the per-scheme FileSystem cache
keys are written by
+ // StorageAdapter itself, and this merge is where several storages'
maps meet.
+ Map<String, String> hdfsProps = new HashMap<>();
+ hdfsProps.put("uri", "hdfs://test/1.orc");
+ hdfsProps.put("hadoop.username", "userA");
+ StorageAdapter hdfs = StorageAdapter.of(hdfsProps);
+ Map<String, String> s3Props = new HashMap<>();
+ s3Props.put("s3.endpoint", "s3.us-west-2.amazonaws.com");
+ s3Props.put("s3.access_key", "ak1");
+ s3Props.put("s3.secret_key", "secret");
+ StorageAdapter s3 = StorageAdapter.of(s3Props);
+
+ Map<StorageTypeId, StorageAdapter> storagePropertiesMap = new
HashMap<>();
+ storagePropertiesMap.put(hdfs.getType(), hdfs);
+ storagePropertiesMap.put(s3.getType(), s3);
+
+ Map<String, String> result =
CredentialUtils.getBackendPropertiesFromStorageMap(storagePropertiesMap);
+
+ // Each storage keeps its own fingerprint under its own schemes; a
single shared key would
+ // let whichever storage merged last decide the cache identity of the
other one.
+ Assertions.assertEquals(hdfs.getFsCacheFingerprint(),
+ result.get(FsCacheKeys.fsCacheKeyProperty("hdfs")));
+ Assertions.assertEquals(s3.getFsCacheFingerprint(),
result.get(FsCacheKeys.fsCacheKeyProperty("s3a")));
+ Assertions.assertNotEquals(hdfs.getFsCacheFingerprint(),
s3.getFsCacheFingerprint());
+ Assertions.assertNull(result.get(FsCacheKeys.FS_CACHE_KEY_PROPERTY));
+ }
+
@Test
public void testGetBackendPropertiesFromStorageMapWithNullValues() {
StorageAdapter s3Properties = Mockito.mock(StorageAdapter.class);
diff --git
a/fe/fe-filesystem/fe-filesystem-cos/src/test/java/org/apache/doris/filesystem/cos/CosFileSystemPropertiesTest.java
b/fe/fe-filesystem/fe-filesystem-cos/src/test/java/org/apache/doris/filesystem/cos/CosFileSystemPropertiesTest.java
index 5bd9508ab99..e21e24a4539 100644
---
a/fe/fe-filesystem/fe-filesystem-cos/src/test/java/org/apache/doris/filesystem/cos/CosFileSystemPropertiesTest.java
+++
b/fe/fe-filesystem/fe-filesystem-cos/src/test/java/org/apache/doris/filesystem/cos/CosFileSystemPropertiesTest.java
@@ -21,6 +21,7 @@ import org.apache.doris.filesystem.FileSystem;
import org.apache.doris.filesystem.FileSystemType;
import org.apache.doris.filesystem.properties.BackendStorageKind;
import org.apache.doris.filesystem.properties.BackendStorageProperties;
+import org.apache.doris.filesystem.properties.FsCacheKeys;
import org.apache.doris.filesystem.properties.StorageKind;
import org.apache.doris.filesystem.spi.S3CompatibleFileSystem;
@@ -29,6 +30,7 @@ import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -158,6 +160,32 @@ class CosFileSystemPropertiesTest {
Assertions.assertEquals("10000",
hadoopKv.get("fs.s3a.connection.timeout"));
}
+ @Test
+ void toHadoopConfigurationMap_keysFileSystemCacheByCredentialFingerprint()
{
+ CosFileSystemProperties properties = CosFileSystemProperties.of(Map.of(
+ "cos.endpoint", "https://cos.ap-guangzhou.myqcloud.com";,
+ "cos.access_key", "ak",
+ "cos.secret_key", "sk"));
+ CosFileSystemProperties otherCredentials =
CosFileSystemProperties.of(Map.of(
+ "cos.endpoint", "https://cos.ap-guangzhou.myqcloud.com";,
+ "cos.access_key", "other-ak",
+ "cos.secret_key", "other-sk"));
+
+ Map<String, String> hadoopKv = properties.toHadoopConfigurationMap();
+
+ // The Hadoop FileSystem cache stays on. Instead of the retired blanket
+ // fs.<scheme>.impl.disable.cache=true, every scheme this storage can
be opened with carries
+ // its credential fingerprint, which the Doris-patched FileSystem
folds into its cache key.
+ for (String scheme : List.of("cos", "cosn", "s3", "s3a")) {
+ Assertions.assertNull(hadoopKv.get("fs." + scheme +
".impl.disable.cache"), scheme);
+ Assertions.assertEquals(properties.fsCacheFingerprint(),
+ hadoopKv.get(FsCacheKeys.fsCacheKeyProperty(scheme)),
scheme);
+ }
+ // Never the shared, scheme-less name: per-scheme names are what keep
a merge lossless.
+ Assertions.assertNull(hadoopKv.get(FsCacheKeys.FS_CACHE_KEY_PROPERTY));
+ Assertions.assertNotEquals(properties.fsCacheFingerprint(),
otherCredentials.fsCacheFingerprint());
+ }
+
@Test
void bind_rejectsPartialStaticCredentialsLikeFeCore() {
IllegalArgumentException exception =
Assertions.assertThrows(IllegalArgumentException.class,
diff --git
a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java
b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java
index d1ac47224e0..2802018a50d 100644
---
a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java
+++
b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java
@@ -17,9 +17,12 @@
package org.apache.doris.filesystem.gcs;
+import org.apache.doris.filesystem.properties.FsCacheKeys;
+
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.util.List;
import java.util.Map;
class GcsFileSystemPropertiesTest {
@@ -125,6 +128,30 @@ class GcsFileSystemPropertiesTest {
Assertions.assertEquals("sk", cfg.get("fs.s3a.secret.key"));
}
+ @Test
+ void toHadoopConfigurationMap_keysFileSystemCacheByCredentialFingerprint()
{
+ GcsFileSystemProperties properties = GcsFileSystemProperties.of(Map.of(
+ "gs.access_key", "ak",
+ "gs.secret_key", "sk"));
+ GcsFileSystemProperties otherCredentials =
GcsFileSystemProperties.of(Map.of(
+ "gs.access_key", "other-ak",
+ "gs.secret_key", "other-sk"));
+
+ Map<String, String> hadoopKv = properties.toHadoopConfigurationMap();
+
+ // The Hadoop FileSystem cache stays on. Instead of the retired blanket
+ // fs.<scheme>.impl.disable.cache=true, every scheme this storage can
be opened with carries
+ // its credential fingerprint, which the Doris-patched FileSystem
folds into its cache key.
+ for (String scheme : List.of("gs", "s3", "s3a")) {
+ Assertions.assertNull(hadoopKv.get("fs." + scheme +
".impl.disable.cache"), scheme);
+ Assertions.assertEquals(properties.fsCacheFingerprint(),
+ hadoopKv.get(FsCacheKeys.fsCacheKeyProperty(scheme)),
scheme);
+ }
+ // Never the shared, scheme-less name: per-scheme names are what keep
a merge lossless.
+ Assertions.assertNull(hadoopKv.get(FsCacheKeys.FS_CACHE_KEY_PROPERTY));
+ Assertions.assertNotEquals(properties.fsCacheFingerprint(),
otherCredentials.fsCacheFingerprint());
+ }
+
@Test
void of_rejectsInvalidUsePathStyle() {
IllegalArgumentException e =
Assertions.assertThrows(IllegalArgumentException.class,
diff --git
a/fe/fe-filesystem/fe-filesystem-obs/src/test/java/org/apache/doris/filesystem/obs/ObsFileSystemPropertiesTest.java
b/fe/fe-filesystem/fe-filesystem-obs/src/test/java/org/apache/doris/filesystem/obs/ObsFileSystemPropertiesTest.java
index a9f288e5f6b..1584a84d7e5 100644
---
a/fe/fe-filesystem/fe-filesystem-obs/src/test/java/org/apache/doris/filesystem/obs/ObsFileSystemPropertiesTest.java
+++
b/fe/fe-filesystem/fe-filesystem-obs/src/test/java/org/apache/doris/filesystem/obs/ObsFileSystemPropertiesTest.java
@@ -21,6 +21,7 @@ import org.apache.doris.filesystem.FileSystem;
import org.apache.doris.filesystem.FileSystemType;
import org.apache.doris.filesystem.properties.BackendStorageKind;
import org.apache.doris.filesystem.properties.BackendStorageProperties;
+import org.apache.doris.filesystem.properties.FsCacheKeys;
import org.apache.doris.filesystem.properties.StorageKind;
import org.apache.doris.filesystem.spi.S3CompatibleFileSystem;
@@ -29,6 +30,7 @@ import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -129,6 +131,32 @@ class ObsFileSystemPropertiesTest {
Assertions.assertEquals("10000",
hadoopKv.get("fs.s3a.connection.timeout"));
}
+ @Test
+ void toHadoopConfigurationMap_keysFileSystemCacheByCredentialFingerprint()
{
+ ObsFileSystemProperties properties = ObsFileSystemProperties.of(Map.of(
+ "obs.endpoint", "https://obs.cn-north-4.myhuaweicloud.com";,
+ "obs.access_key", "ak",
+ "obs.secret_key", "sk"));
+ ObsFileSystemProperties otherCredentials =
ObsFileSystemProperties.of(Map.of(
+ "obs.endpoint", "https://obs.cn-north-4.myhuaweicloud.com";,
+ "obs.access_key", "other-ak",
+ "obs.secret_key", "other-sk"));
+
+ Map<String, String> hadoopKv = properties.toHadoopConfigurationMap();
+
+ // The Hadoop FileSystem cache stays on. Instead of the retired blanket
+ // fs.<scheme>.impl.disable.cache=true, every scheme this storage can
be opened with carries
+ // its credential fingerprint, which the Doris-patched FileSystem
folds into its cache key.
+ for (String scheme : List.of("obs", "s3", "s3a")) {
+ Assertions.assertNull(hadoopKv.get("fs." + scheme +
".impl.disable.cache"), scheme);
+ Assertions.assertEquals(properties.fsCacheFingerprint(),
+ hadoopKv.get(FsCacheKeys.fsCacheKeyProperty(scheme)),
scheme);
+ }
+ // Never the shared, scheme-less name: per-scheme names are what keep
a merge lossless.
+ Assertions.assertNull(hadoopKv.get(FsCacheKeys.FS_CACHE_KEY_PROPERTY));
+ Assertions.assertNotEquals(properties.fsCacheFingerprint(),
otherCredentials.fsCacheFingerprint());
+ }
+
@Test
void hadoopMap_selectsObsFileSystemWithoutLinkingIt() {
// Premise this test rests on, and the exact shape of a deployment
where fe-core no longer
diff --git
a/fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssFileSystemPropertiesTest.java
b/fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssFileSystemPropertiesTest.java
index 12116f65536..d2975a6b606 100644
---
a/fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssFileSystemPropertiesTest.java
+++
b/fe/fe-filesystem/fe-filesystem-oss/src/test/java/org/apache/doris/filesystem/oss/OssFileSystemPropertiesTest.java
@@ -21,6 +21,7 @@ import org.apache.doris.filesystem.FileSystem;
import org.apache.doris.filesystem.FileSystemType;
import org.apache.doris.filesystem.properties.BackendStorageKind;
import org.apache.doris.filesystem.properties.BackendStorageProperties;
+import org.apache.doris.filesystem.properties.FsCacheKeys;
import org.apache.doris.filesystem.properties.StorageKind;
import org.apache.doris.filesystem.spi.S3CompatibleFileSystem;
@@ -30,6 +31,7 @@ import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.lang.reflect.Method;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -202,6 +204,32 @@ class OssFileSystemPropertiesTest {
Assertions.assertEquals("10000",
hadoopKv.get("fs.s3a.connection.timeout"));
}
+ @Test
+ void toHadoopConfigurationMap_keysFileSystemCacheByCredentialFingerprint()
{
+ OssFileSystemProperties properties = OssFileSystemProperties.of(Map.of(
+ "oss.endpoint", "https://oss-cn-hangzhou.aliyuncs.com";,
+ "oss.access_key", "ak",
+ "oss.secret_key", "sk"));
+ OssFileSystemProperties otherCredentials =
OssFileSystemProperties.of(Map.of(
+ "oss.endpoint", "https://oss-cn-hangzhou.aliyuncs.com";,
+ "oss.access_key", "other-ak",
+ "oss.secret_key", "other-sk"));
+
+ Map<String, String> hadoopKv = properties.toHadoopConfigurationMap();
+
+ // The Hadoop FileSystem cache stays on. Instead of the retired blanket
+ // fs.<scheme>.impl.disable.cache=true, every scheme this storage can
be opened with carries
+ // its credential fingerprint, which the Doris-patched FileSystem
folds into its cache key.
+ for (String scheme : List.of("oss", "s3", "s3a")) {
+ Assertions.assertNull(hadoopKv.get("fs." + scheme +
".impl.disable.cache"), scheme);
+ Assertions.assertEquals(properties.fsCacheFingerprint(),
+ hadoopKv.get(FsCacheKeys.fsCacheKeyProperty(scheme)),
scheme);
+ }
+ // Never the shared, scheme-less name: per-scheme names are what keep
a merge lossless.
+ Assertions.assertNull(hadoopKv.get(FsCacheKeys.FS_CACHE_KEY_PROPERTY));
+ Assertions.assertNotEquals(properties.fsCacheFingerprint(),
otherCredentials.fsCacheFingerprint());
+ }
+
@Test
void bind_rejectsPartialStaticCredentialsLikeFeCore() {
IllegalArgumentException exception =
Assertions.assertThrows(IllegalArgumentException.class,
diff --git
a/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java
b/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java
index a081233add1..46e8387c4d2 100644
---
a/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java
+++
b/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.filesystem.s3;
import org.apache.doris.filesystem.properties.BackendStorageKind;
import org.apache.doris.filesystem.properties.BackendStorageProperties;
+import org.apache.doris.filesystem.properties.FsCacheKeys;
import org.apache.doris.filesystem.properties.HadoopStorageProperties;
import org.junit.jupiter.api.Assertions;
@@ -26,6 +27,7 @@ import org.junit.jupiter.api.Test;
import
software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
class S3FileSystemPropertiesTest {
@@ -256,6 +258,32 @@ class S3FileSystemPropertiesTest {
Assertions.assertEquals("true",
hadoopMap.get("fs.s3a.path.style.access"));
}
+ @Test
+ void toHadoopConfigurationMap_keysFileSystemCacheByCredentialFingerprint()
{
+ S3FileSystemProperties properties = S3FileSystemProperties.of(Map.of(
+ "s3.endpoint", "https://s3.us-west-2.amazonaws.com";,
+ "s3.access_key", "ak",
+ "s3.secret_key", "sk"));
+ S3FileSystemProperties otherCredentials =
S3FileSystemProperties.of(Map.of(
+ "s3.endpoint", "https://s3.us-west-2.amazonaws.com";,
+ "s3.access_key", "other-ak",
+ "s3.secret_key", "other-sk"));
+
+ Map<String, String> hadoopKv = properties.toHadoopConfigurationMap();
+
+ // The Hadoop FileSystem cache stays on. Instead of the retired blanket
+ // fs.<scheme>.impl.disable.cache=true, every scheme this storage can
be opened with carries
+ // its credential fingerprint, which the Doris-patched FileSystem
folds into its cache key.
+ for (String scheme : List.of("s3", "s3a", "s3n")) {
+ Assertions.assertNull(hadoopKv.get("fs." + scheme +
".impl.disable.cache"), scheme);
+ Assertions.assertEquals(properties.fsCacheFingerprint(),
+ hadoopKv.get(FsCacheKeys.fsCacheKeyProperty(scheme)),
scheme);
+ }
+ // Never the shared, scheme-less name: per-scheme names are what keep
a merge lossless.
+ Assertions.assertNull(hadoopKv.get(FsCacheKeys.FS_CACHE_KEY_PROPERTY));
+ Assertions.assertNotEquals(properties.fsCacheFingerprint(),
otherCredentials.fsCacheFingerprint());
+ }
+
@Test
void of_bindsAndNormalizesCredentialsProviderType() {
Map<String, String> raw = new HashMap<>();
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
