nsivarajan opened a new pull request, #65795: URL: https://github.com/apache/doris/pull/65795
### What problem does this PR solve? Issue Number: close #65793 Related PR: part 3 of 4 — PR-1 https://github.com/apache/doris/pull/61329 | PR-2 #65794 | PR-4 #xxx ## Problem Summary: Doris BE currently accesses OSS through the AWS S3-compatible SDK path. This works for basic AK/SK operations but cannot use ECS instance profile credentials, STS AssumeRole, RRSA, or environment variable providers — all of which require the native Alibaba Cloud OSS C++ SDK. This PR implements the complete native OSS filesystem stack in BE, routed via a new `enable_oss_native_sdk` config flag (default `false`). The existing S3-compatible path is fully preserved and used when the flag is off. **Three routing paths all now emit `"provider"="OSS"` to BE:** | Path | Source | FE class | |---|---|---| | Catalog / SPI | Thrift property map | `OssFileSystemProperties.toBackendKv()` | | Storage Vault | `ObjectStoreInfoPB` | `OSSConf::get_oss_conf()` | | S3 TVF / Resource | Thrift property map | `OSSProperties.getBackendConfigProperties()` | --- Changes **New shared headers (`common/cpp/`)** - `oss_common.h` — `OSSCredProviderType` enum, `normalize_oss_endpoint()` - `oss_credential_provider.h/cpp` — `ECSMetadataCredentialsProvider` (metadata service, auto-refresh), `OSSSTSCredentialProvider` (STS AssumeRole, refresh 5 min before expiry), `OSSDefaultCredentialsProvider` (env → config → ECS chain) **New BE filesystem (`be/src/io/fs/`)** - `oss_v2_obj_storage_client.h/cpp` — `OSSv2ObjStorageClient : ObjStorageClient`; implements all object operations using native OSS SDK - `oss_file_system.h/cpp` — `OSSFileSystem : RemoteFileSystem`; thread-safe client holder with `shared_mutex`; prefix normalised in `create()` factory before parent constructor; multipart copy for renames >1 GB - `oss_file_reader.h/cpp` — range reads with exponential backoff + jitter on 429/SlowDown errors - `oss_file_writer.h/cpp` — multipart upload with async close **New BE utility (`be/src/util/`)** - `oss_util.h/cpp` — `OSSClientConf`, `OSSConf`, `OSSClientFactory` singleton; `get_oss_conf(ObjectStoreInfoPB)` for storage vault path; SDK initialised once via `std::call_once` **Modified BE files (additive)** - `be/src/io/fs/file_system.h` — `FileSystemType::OSS` added to enum - `be/src/util/s3_util.cpp` — `_create_oss_client()` under `#ifdef USE_OSS`; `convert_properties_to_s3_conf()` detects `"provider"="OSS"` → `ObjStorageType::OSS`; `S3ClientFactory::create()` routes to native OSS client when `enable_oss_native_sdk=true` - `be/src/common/config.h/cpp` — `enable_oss_native_sdk=false`, `max_oss_client_retry`, `oss_read_base_wait_time_ms`, `oss_read_max_wait_time_ms` **Modified FE files (additive)** - `OSSProperties.java` — adds `roleArn`, `externalId`; `getBackendConfigProperties()` emits `"provider"="OSS"`, `AWS_ROLE_ARN`, `AWS_EXTERNAL_ID` - `OssFileSystemProperties.java` — `toBackendKv()` adds `"provider"="OSS"` **Tests** - `be/test/util/s3_util_test.cpp` — 3 tests under `#ifdef USE_OSS`: provider detection, case-insensitive `"oss"`, missing provider defaults to AWS - `fe-core/OSSPropertiesTest.java` — `testBackendConfigIncludesProviderOss()`, `testBackendConfigForwardsRoleArnAndExternalId()` --- Design decisions - **Additive only** — `enable_oss_native_sdk=false` by default; all existing S3-compatible OSS behaviour is unchanged when off - **Prefix normalisation before constructor** — `RemoteFileSystem` parent reads the prefix in its constructor; normalisation must happen in `OSSFileSystem::create()` factory before `new OSSFileSystem(...)` is called - **Thread safety** — `OssClient` is thread-safe; `shared_mutex` in `OSSClientHolder` guards client pointer replacement only, not each call - **Credential caching** — all three custom providers cache credentials and refresh 5 minutes before expiry to avoid mid-request token expiry ### Release note [be] Add native Alibaba Cloud OSS filesystem (`OSSFileSystem`, `OSSFileReader`, `OSSFileWriter`) using the official OSS C++ SDK. Activated by `enable_oss_native_sdk=true` in `be.conf` (default `false`). Supports ECS instance profile, STS AssumeRole, RRSA, env vars, and static AK/SK. Existing S3-compatible OSS path unchanged when flag is off. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [X] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [X] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [X] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
