Doris-Breakwater commented on issue #66554: URL: https://github.com/apache/doris/issues/66554#issuecomment-5205320019
Breakwater-GitHub-Analysis-Slot: slot_fc3e45704055 ### Initial triage **Judgment: valid BE credential-provider compatibility bug; high confidence.** The issue is open with no labels, assignee, milestone, or earlier comments, and the author is willing to submit a PR. The unsupported behavior is present at the current remote heads of both `master` (`8e4bd4d5f11d68ddd5fd4ae2ee153ad8d6779323`) and `branch-4.1` (`e48fc7e97b98909199d27bfd3aba3924f5588c51`). ### Verified facts - Both branches still vendor [AWS SDK for C++ 1.11.219](https://github.com/apache/doris/blob/8e4bd4d5f11d68ddd5fd4ae2ee153ad8d6779323/thirdparty/vars.sh#L382-L384). AWS lists [1.11.263 as the minimum C++ SDK version for EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html). - EKS Pod Identity injects `AWS_CONTAINER_CREDENTIALS_FULL_URI` and `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE`; the projected token file is refreshed by EKS. This is the documented [Pod Identity environment and volume setup](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html). - Doris's [`CustomAwsCredentialsProviderChain`](https://github.com/apache/doris/blob/8e4bd4d5f11d68ddd5fd4ae2ee153ad8d6779323/common/cpp/custom_aws_credentials_provider_chain.cpp#L33-L80) recognizes the full URI but only reads `AWS_CONTAINER_AUTHORIZATION_TOKEN`, once while constructing `TaskRoleCredentialsProvider`. It never reads `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE`. - In SDK 1.11.219, `TaskRoleCredentialsProvider::Reload()` refreshes the returned temporary AWS credentials, but its `ECSCredentialsClient` retains the authorization token supplied at construction. Therefore a token copied into `AWS_CONTAINER_AUTHORIZATION_TOKEN` at BE startup becomes stale after the projected file rotates. - Upstream SDK 1.11.263 added the intended mechanism: the default chain passes the token-file path into `GeneralHTTPCredentialsProvider`, and [`Reload()` reads that file again before each credential-endpoint request](https://github.com/aws/aws-sdk-cpp/blob/1.11.263/src/aws-cpp-sdk-core/source/auth/GeneralHTTPCredentialsProvider.cpp#L214-L231). - There is a second affected route: an explicitly selected [`CredProviderType::Container`](https://github.com/apache/doris/blob/8e4bd4d5f11d68ddd5fd4ae2ee153ad8d6779323/be/src/util/s3_util.cpp#L358-L360) reads only `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`. EKS supplies the full URI, so this route also needs coverage. The cloud recycler has the same construction pattern. One detail of the reported timeline is not yet explained. With the standard EKS injection there is no raw `AWS_CONTAINER_AUTHORIZATION_TOKEN`, so the current code can fail on the **first** credential fetch, not only after rotation. The stated "works initially, then fails" sequence is consistent if an entrypoint or another component copied the initial file contents into the raw-token environment variable, or if another credential source was initially selected. This does not weaken the confirmed missing token-file support, but it should be clarified before treating the exact timeline as reproduced. ### Information requested from the reporter 1. Exact Doris build/commit for each tested version, plus the BE value of `aws_credentials_provider_version` and the catalog's `s3.credentials_provider_type` (or confirmation that it is unset/default). 2. Presence and paths of the `AWS_CONTAINER_*` variables inside the BE container. Please redact the authorization token itself; only indicate whether the raw-token variable is present and how it is populated. 3. The exact first failure and surrounding BE credential-provider/S3 log lines, including timestamps relative to the credential expiry and projected-token rotation. Please redact credentials, account IDs, bucket names, and other sensitive values. 4. Confirmation that the first S3 query really used the Pod Identity role rather than a node role, profile, or another earlier provider in the chain. These details are useful to reconcile the timing, but they are not blockers to fixing the verified incompatibility. ### Recommended implementation and validation - Prefer upgrading the C++ SDK to at least 1.11.263 (or a newer vetted 1.11.x release) and porting the upstream general/container HTTP provider logic into `CustomAwsCredentialsProviderChain` while preserving Doris's intentional provider ordering. **An SDK bump alone is insufficient** because the custom chain currently bypasses the new default-chain token-file logic. - Apply the same helper/provider construction to explicit `CredProviderType::Container` paths in BE and cloud code. Audit other C++ container-credential call sites so behavior is consistent. - Add a focused unit test with a loopback credential endpoint and a temporary authorization-token file: accept token A for the first short-lived credential response, replace the file with token B, force/await provider refresh, and verify the next request sends B and returns refreshed credentials. Also retain coverage for ECS relative URI and the direct `AWS_CONTAINER_AUTHORIZATION_TOKEN` form. - After the master fix is verified, backport it to 4.1. Restarting the BE pod is only a temporary operational workaround because it reloads the initial token but does not make rotation safe. Suggested triage: `kind/fix` plus the repository's BE/object-storage label, and assign the S3 credential-provider owner. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
