thedevopsmachine opened a new issue, #14100:
URL: https://github.com/apache/apisix/issues/14100
### Current Behavior
The AWS Secrets Manager secret backend (`apisix/secret/aws.lua`) treats all
credentials as **static strings**. On each request it resolves `access_key_id`,
`secret_access_key`, and `session_token` from either an env-var reference or
the literal config value, and passes them directly into a fixed `Credentials`
object:
```lua
local access_key_id = env.fetch_by_uri(conf.access_key_id) or
conf.access_key_id
local secret_access_key = env.fetch_by_uri(conf.secret_access_key) or
conf.secret_access_key
local session_token = env.fetch_by_uri(conf.session_token) or
conf.session_token
local credentials = aws_instance:Credentials({
accessKeyId = access_key_id,
secretAccessKey = secret_access_key,
sessionToken = session_token,
})
```
Because these are passed as literal values, there is no credential
**provider** behind them — nothing tracks expiration and nothing re-fetches
credentials when they near expiry.
This is a problem specifically for **temporary credentials**, which are the
only kind that carry a `session_token`. Temporary credentials
(STS/assumed-role, IRSA/web-identity, ECS/EKS task roles, EC2 instance
profiles) expire — typically after 1 hour, up to the role's max session
duration. Once they expire, every `GetSecretValue` call fails with an
authentication error, and the only way to recover is to rewrite the
`session_token` (and key id / secret) in the env var or config and reload
APISIX. In practice this would require re-plumbing the credentials roughly
every hour, which is not operationally viable for a long-running gateway.
Two compounding factors:
1. `require("resty.aws.config")` reads the AWS environment at module init
(worker start). A session token exported into the environment is captured as a
snapshot at that point; rotating the underlying env var at the OS level is not
reliably picked up by an already-running worker without a reload.
2. The schema marks `access_key_id` and `secret_access_key` as `required`,
and `make_request_to_aws` *always* constructs an explicit `Credentials{}` from
whatever it resolved. There is currently **no way** to tell the module "don't
use static credentials — resolve them from the default AWS credential provider
chain instead." That provider chain is exactly what would discover an ECS/EKS
task role, IRSA/web-identity, or instance profile and **auto-refresh** the
temporary credentials (including the session token) before they expire.
Note: using the backend with **long-term IAM user keys** (access key +
secret, no session token) works fine indefinitely — the issue is strictly the
temporary-credential / session-token path.
### Interaction with standalone (file-driven) reload
This is worth calling out because it widens the blast radius in the
file-driven standalone deployment model (`apisix/core/config_yaml.lua`).
In file-driven standalone mode, the config "version" used throughout the
loader is the **file's modification time** (`apisix_yaml_mtime`), which is
global to the whole `apisix.yaml` — there is no per-route or per-collection
revision. Any edit to the file bumps the mtime, and that single mtime becomes
the `conf_version` for **every** config type, including `/secrets`.
The secret value cache in `apisix/secret.lua` is an LRU keyed on the
reference string **plus** `secrets.conf_version`:
```lua
local version = secrets and secrets.conf_version or ""
return secrets_cache(uri, version, fetch_by_uri, uri)
```
Because `secrets.conf_version` in file mode is the global file mtime,
**editing anything in** `apisix.yaml` **rotates the cache key for every secret
reference**, so each referenced secret is re-resolved from AWS Secrets Manager
once on next use — even for routes that did not change. (This differs from the
etcd / API-driven model, where `/secrets` carries its own revision that only
bumps when secret objects actually change, limiting the blast radius. The
API-driven standalone path also retains unchanged items by ID, whereas the file
path rebuilds the whole value list — `self.values = nil` — on any mtime change.)
Consequence for this bug: a single config reload will force re-resolution of
**all** referenced secrets across the gateway. If the underlying credentials
are the broken static/expired session token described above, the failure
surfaces simultaneously across every secret reference on the next reload — not
just on a route that happened to change. It also means the 300s value-cache TTL
is not the only re-fetch trigger in file mode; every file edit is an
additional, gateway-wide trigger.
### Expected Behavior
When explicit `access_key_id` / `secret_access_key` are **not** configured,
`aws.lua` should omit the explicit `Credentials{}` object and let
`lua-resty-aws` resolve credentials via its default credential provider chain,
so that temporary credentials obtained from an instance profile / ECS or EKS
task role / IRSA web-identity are used and **auto-refreshed** before expiry —
with no config churn. This is the behavior operators expect when running APISIX
on ECS/Fargate, EKS, or EC2.
At minimum, the limitation should be documented: the current session-token
field only works with externally-rotated credentials plus a reload, and cannot
auto-refresh.
### Error Logs
```
failed to retrieve data from aws secret manager: invalid status code 400,
{"__type":"ExpiredTokenException","message":"The security token included in the
request is expired"}
```
### Steps to Reproduce
This reproduction uses **file-driven standalone mode** (`deployment.role:
data_plane`, `deployment.role_data_plane.config_provider: yaml`). The thing
that makes the no-refresh bug observable is that temporary credentials are only
read when a secret reference is *resolved*, and a resolved value is then cached
in the `apisix/secret.lua` LRU (keyed on the reference string + the `/secrets`
conf\_version). Since most secrets are resolved once at startup and then served
warm, the expired session token stays hidden until something forces a
re-resolution. A config-file edit is the clean, deterministic trigger: in file
mode the version token is the **global file mtime**, so saving any edit rotates
the cache key for every secret reference and forces re-resolution against AWS —
even for routes that did not change.
1. Run APISIX on a host/container whose only AWS credentials are
**temporary** — an EC2 instance profile, ECS/EKS task role, IRSA web-identity,
or a manual STS session. For a deterministic repro, mint a short session and
export it so expiry is observable:
```bash
# ~15 min session so you don't wait an hour; STS max for this call is
governed by the role
eval "$(aws sts assume-role \
--role-arn arn:aws:iam::<acct>:role/apisix-secrets-reader \
--role-session-name apisix-repro \
--duration-seconds 900 \
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
--output text | awk '{print "export AWS_ACCESS_KEY_ID="$1"\nexport
AWS_SECRET_ACCESS_KEY="$2"\nexport AWS_SESSION_TOKEN="$3}')"
```
2. Create a secret in AWS Secrets Manager, e.g. `repro` with value
`{"password":"p1"}`.
3. In `conf/config.yaml`, select file-driven standalone mode:
```yaml
deployment:
role: data_plane
role_data_plane:
config_provider: yaml
```
4. In `conf/apisix.yaml`, define the AWS secret manager object (pointing at
the temporary creds via `$env://`) and a route that references the secret:
```yaml
secrets:
- id: aws/1
access_key_id: $env://AWS_ACCESS_KEY_ID
secret_access_key: $env://AWS_SECRET_ACCESS_KEY
session_token: $env://AWS_SESSION_TOKEN
region: us-east-1
routes:
- id: r1
uri: /r1
plugins:
proxy-rewrite:
headers:
set:
X-DB-Pass: "$secret://aws/1/repro"
upstream: { nodes: { "127.0.0.1:1980": 1 }, type: roundrobin }
#END
```
5. Start APISIX and send one request to `/r1`. It succeeds — the secret is
resolved and the resolved value is now cached in the LRU. This is the **only**
window in which the temporary credentials are read.
6. Let the assumed-role session from step 1 expire (the 15-min session
above, or ~1h for a default role session). Nothing fails yet: the warm cache
entry keeps serving the previously resolved value, so the gateway appears
healthy even though the underlying credentials are already dead.
7. Make a **trivially small, unrelated** edit to `conf/apisix.yaml` — e.g.
add a second route `/r2` that does **not** reference any secret, or change a
comment — and save (keeping the `#END` marker). This bumps the file's
modification time. Crucially, the edit does **not** touch `/r1` or the
`aws/repro` secret object.
8. The standalone loader polls the file mtime (~1s), sees the change, and
rebuilds the in-memory config. Because the version token in file mode is the
**global file mtime**, the `/secrets` conf\_version changes too, which rotates
the LRU cache key for **every** secret reference — including the untouched
`/r1`.
9. Send a request to `/r1` — a route you did **not** edit. The rotated
cache key forces a cache miss, so `secret.lua` re-invokes `aws.lua`, which
rebuilds the static `Credentials{}` from the now-expired `$env://` session
token and calls `GetSecretValue`.
**Result:** the request to the **unchanged** route `/r1` fails with an
expired-token error (see Error Logs) immediately after the unrelated config
edit. The credentials are never refreshed — every subsequent re-resolution
reuses the same frozen, expired env values. Recovery requires rewriting the
credentials in env/config and restarting the worker (so the
`require("resty.aws.config")` env snapshot is re-read). There is no
configuration that makes the backend source auto-refreshing temporary
credentials from the standard AWS provider chain. Note the blast radius:
because the file-mode version token is the global file mtime, the edit
invalidates the cache key for **every** secret reference, so in a gateway with
many secret-referencing routes an expired session token takes them all down at
once on the next reload — not just a route that changed.
> Note: the config edit above is just a deterministic way to force
re-resolution. The same failure occurs without any edit once a warm cache entry
passes its TTL (`apisix.lru.secret.ttl`, default 300s) and the next request
re-resolves it — the config edit simply triggers that cache miss on demand
rather than waiting out the TTL.
### Environment
- APISIX version (run `apisix version`): 3.19
- Operating system (run `uname -a`): Amazon Linux 2023
6.18.54-121.166.amzn2023.x86_64
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
