[
https://issues.apache.org/jira/browse/KNOX-3512?focusedWorklogId=1047239&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-1047239
]
ASF GitHub Bot logged work on KNOX-3512:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Oct/26 12:36
Start Date: 13/Oct/26 12:36
Worklog Time Spent: 10m
Work Description: smolnar82 opened a new pull request, #1451:
URL: https://github.com/apache/knox/pull/1451
[KNOX-3512](https://issues.apache.org/jira/browse/KNOX-3512) - Per-issuer CA
bundle (jwksCa) for trusted OIDC issuers
## What changes were proposed in this pull request?
Today, Knox fetches a trusted OIDC issuer's discovery document and JWKS
using the JVM default truststore. To trust a cluster whose API server uses a
self-signed CA (e.g. a k8s/k3s issuer), you have to import that CA into the
global truststore. That widens trust for every other TLS client in the gateway,
and every CA rotation needs a restart.
This PR adds an optional, per-issuer **`jwksCa`** to the trusted OIDC issuer
registry:
- **Two forms**
- inline PEM bundle (≤ 65,000 chars)
- Kubernetes Secret reference `secret:[<namespace>/]<name>#<key>`. The
namespace defaults to `knox`; name and key are required.
- **Pinned trust:** when set, that issuer's discovery and JWKS connections
trust *only* that bundle. The global truststore is never consulted or modified.
Hostname verification stays on.
- **Fail-closed:** if the issuer has a `jwksCa` that cannot be loaded
(missing Secret or key, bad PEM), its tokens are rejected (401). Knox never
falls back to the default truststore.
- **Unchanged when unset:** issuers without a `jwksCa` behave exactly as
before.
- **No restart needed for CA changes**
- Inline bundle: re-registering (delete + create, as the operator does)
applies immediately.
- Secret reference: the Secret is read lazily on first use, with no
background refresh. It is re-read:
- after `gateway.trusted.oidc.issuer.ca.secret.max.age.secs` (default
`1800`)
- on `POST .../trusted-oidc-issuers/refresh-jwks`
- after a TLS trust failure on a discovery or JWKS fetch. Knox then
retries the fetch once. This is rate-limited to once per 30s per issuer and is
not triggered by timeouts, refused connections or non-200 responses.
A re-read with unchanged content keeps the existing SSL context. A
failed periodic re-read keeps the last good bundle.
### Changes by area
- **Model / persistence** (`gateway-spi`, `gateway-server`)
- `TrustedOidcIssuer` gains `jwksCa`.
- A new `jwks_ca` column: `TEXT`, and `CLOB` on Oracle. It is added in the
CREATE scripts only, because the table is unreleased.
- New `JwksCa` utility: secret-reference parsing and PEM bundle validation.
- **REST API** (`gateway-service-knoxidf`)
- `POST` accepts `jwksCa`; `GET` returns it. CA certificates are public,
and the operator needs the value to detect drift.
- Returns `400 invalid_request` when:
- `jwksCa` is set with `dynamicJwks=false`
- the value is oversized
- a secret reference has bad syntax
- an inline bundle has no parseable X.509 certificate
The syntax check alone is enough to register a Secret reference; the
Secret itself may be created later.
- **Trust material** (`IssuerTrustProvider`)
- Builds an `SSLContext` from an in-memory keystore holding only the
bundle's certificates.
- Reads Secrets through the fabric8 client (the same kubeconfig/in-cluster
config as the ServiceAccount validator).
- Logs SHA-256 fingerprints only, never PEM or Secret contents.
- **Discovery and JWKS fetch**
- `OIDCDiscoveryHelper` uses a pinned HTTP client for issuers that have a
`jwksCa`.
- `DefaultTokenAuthorityService` builds the Nimbus `JWKSource` with a new
`IssuerTrustResourceRetriever` that wraps a `DefaultResourceRetriever` with the
issuer's socket factory. On a TLS trust failure it performs the Secret re-read
and single retry. The re-read lives here because Nimbus's retrying and
outage-tolerant layers can swallow the TLS cause.
- **SPI changes**
- `TrustedOidcIssuerService#getJwksTrust(String)`: new method returning
`Optional<IssuerJwksTrust>`.
- `JWTokenAuthority#verifyToken(..., IssuerJwksTrust)`: new default
method, so existing implementations still compile.
- `AbstractJWTFilter#resolveRegisteredIssuerJwks` now returns a
`RegisteredIssuerJwks(uris, jwksTrust)` record instead of `Set<URI>`. The URIs
and trust come from a single lookup, so they always belong together.
- **Signing-key rotation:** works without any config change. A token whose
`kid` is unknown makes the JWK source re-fetch the JWKS.
- **Config:** `gateway.trusted.oidc.issuer.ca.secret.max.age.secs` (default
`1800`).
- **Docs** (`knox-site/docs/knoxidf/`)
- `endpoints.md`: the new "Per-issuer CA bundle (`jwksCa`)" section covers
both forms, the 400 rules and the reload behaviour.
- `configuration.md`: the new property.
- `security.md`: new "Trusting self-signed issuer CAs" section. It covers:
- pinning
- the RBAC Knox's ServiceAccount needs: `get` on `secrets` in each
referenced namespace
- the shared-root option
- the HA caveat
- `federation.md`: a pointer to the new section.
### Follow-ups (other repos)
- knox-operator: add `jwksCa` to the `TrustedOidcIssuer` CRD and include it
in the drift check, so a CA change triggers delete + create.
- knox-helm: add a Role/RoleBinding granting the Knox ServiceAccount `get`
on Secrets in the referenced namespaces. This replaces the cacerts import and
the restart-on-rotation approach.
## How was this patch tested?
Unit and integration tests, run with `mvn install` on `gateway-util-common`,
`gateway-spi`, `gateway-spi-common`, `gateway-provider-security-jwt`,
`gateway-server` and `gateway-service-knoxidf`. Checkstyle, PMD and SpotBugs
all pass. Totals: gateway-spi 364, gateway-provider-security-jwt 578,
gateway-server 889 (1 skipped), gateway-service-knoxidf 246, all green. The
downstream modules (`gateway-service-knoxsso`, `gateway-service-knoxtoken`)
also compile.
New and extended tests:
- `JwksCaTest`: secret-reference parsing (with and without a namespace;
malformed values) and PEM bundles with 1 certificate, 2 certificates, or
garbage.
- `IssuerTrustProviderTest`: uses a stubbed Secret reader and an injectable
clock. Covers:
- the Secret is read once on first use and not again before max-age
- re-read after max-age keeps the same SSL context when the content is
unchanged and swaps it when it changed
- re-reads after TLS failures are rate-limited to one per 30s
- a missing Secret or key fails closed
- a failed periodic re-read keeps the old bundle
- the fabric8 reader handles `data` and `stringData`
- `IssuerTrustResourceRetrieverTest`: a TLS trust failure leads to one
re-read and one retry; a timeout or 404 does not.
- `IssuerJwksCaAcceptanceTest`: real Jetty HTTPS JWKS servers with two
independent self-signed CAs. Covers:
- issuers A and B each verify with their own CA
- CA rotation works without a restart
- A's CA cannot reach B, and neither the default truststore nor a plain
`HttpsURLConnection` trusts them, so no trust leaks
- after signing-key rotation, a token with a new `kid` verifies with no
config change
- `OIDCDiscoveryHelperTest`, `JdbcTrustedOidcIssuerServiceTest` (H2
round-trip, `getJwksTrust`, invalidation on register/deregister/refresh),
`TrustedOidcIssuersSchemaTest`, `TrustedOidcIssuersResourceTest` (the 400 cases
and list output), `JWTFederationFilterTokenExchangeTest` (trust passed through,
and a load failure leads to 401).
## Integration Tests
**New suite: `.github/workflows/tests/test_k8s_jwks_ca.py`** (11 tests). It
runs against the throwaway k3s cluster in the compose stack, a real OIDC issuer
at `https://k3s:6443` with a self-signed CA, and covers:
- invalid `jwksCa` values are rejected at registration with 400 and nothing
is stored
- inline CA: the exchange is accepted; with an unrelated CA it is rejected
(pinned)
- the CA does not leak into the default truststore: once the issuer has been
trusted through its `jwksCa`, the same issuer registered without one is still
rejected
- re-registering an inline CA applies without a restart
- a Secret reference works in the default (`knox`) namespace and in an
explicitly named one
- a missing Secret fails closed and works once the Secret is created and
`refresh-jwks` is called
- a missing Secret key fails closed
- a rotated Secret applies on `refresh-jwks`
- a rotated Secret alone also applies, after a TLS failure triggers the
re-read
**CI change:** the k3s CA is **no longer imported into the JVM cacerts**.
The custom `gateway-k8s-delegation.sh` entrypoint did nothing else, so it is
removed and the knox service runs plain `/gateway.sh` again.
`test_k8s_delegation.py` now registers the k3s issuer with its CA as `jwksCa`.
As a result, every positive k8s exchange in CI goes through the pinned
per-issuer path, including the AC4/AC5 delegation tests that verify real
signatures. Supporting fixture: a `knox` namespace manifest
(`k8s/manifests/knox-namespace.yaml`).
Result of the full compose run: `pylint` 10.00/10, **136 passed**:
```
tests-1 | ------------------------------------
tests-1 | Your code has been rated at 10.00/10
tests-1 |
tests-1 | Waiting for knox...
tests-1 | ============================= test session starts
==============================
tests-1 | platform linux
Issue Time Tracking
-------------------
Worklog Id: (was: 1047239)
Remaining Estimate: 0h
Time Spent: 10m
> KnoxIDF: per-issuer CA bundle (jwksCa) for trusted OIDC issuer discovery and
> JWKS fetches
> -----------------------------------------------------------------------------------------
>
> Key: KNOX-3512
> URL: https://issues.apache.org/jira/browse/KNOX-3512
> Project: Apache Knox
> Issue Type: Task
> Components: JWT, Server
> Affects Versions: 3.1.0
> Reporter: Sandor Molnar
> Assignee: Sandor Molnar
> Priority: Major
> Fix For: 3.1.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Problem*
> Knox fetches each trusted OIDC issuer's discovery document and JWKS over
> HTTPS using the global JVM/Knox truststore. Kubernetes apiservers acting as
> issuers often use self-signed certificates, each with its own CA. Today the
> only way to trust them is to import every CA into the global truststore at
> startup. That has two costs:
> - Every CA rotation needs a restart.
> - Trust widens for all TLS clients in Knox (LDAP and the rest).
> TrustedOidcIssuer has no way to carry a CA.
> *Proposal*
> Add an optional {{jwksCa}} field to trusted OIDC issuer registrations. It
> takes one of two forms:
> - Inline PEM: one or more {{---{-}{{-}}BEGIN CERTIFICATE{{-}}{-}---}} blocks.
> - Kubernetes Secret reference: {{{}secret:[<namespace>/]<name>#<key>{}}}.
> Namespace defaults to "knox", the other 2 parts are required. Knox reads it
> through the in-cluster Kubernetes API.
> +Behaviour:+
> - When {{jwksCa}} is set, that issuer's discovery and JWKS TLS connections
> trust only the given bundle, and hostname verification stays on. The global
> truststore is never changed, so other TLS clients in Knox do not trust the
> issuer CA.
> - When {{jwksCa}} is not set, nothing changes.
> - {{jwksCa}} is only accepted together with {{{}dynamicJwks: true{}}}.
> - If {{jwksCa}} is configured but cannot be loaded (Secret or key missing,
> unparsable PEM), verification fails closed and does not fall back to the
> global truststore.
> - CA changes take effect without a restart:
> -- Inline: re-registering the issuer replaces its trust material straight
> away.
> -- Secret reference: Knox reads the Secret lazily, with no background
> refresh. It reads it:
> --- on the first discovery or JWKS fetch for the issuer.
> --- on access, when the cached CA is older than
> {{gateway.trusted.oidc.issuer.ca.secret.max.age.secs}} (default
> {{{}1800{}}}). This picks up CAs removed from the bundle.
> --- after a fetch fails with a TLS trust error (handshake or certificate
> validation). Knox re-reads the Secret and retries once, at most once every
> 30s per issuer. Timeouts and non-200 responses don't trigger a re-read.
> --- on {{{}POST /trusted-oidc-issuers/refresh-jwks{}}}, which drops the
> cached CA so the next fetch reads it again.
> -- If the Secret can't be read the first time, verification fails closed. If
> a max-age re-read fails, Knox keeps the last working CA and logs the failure.
> - Rotating the issuer's signing key still needs no config change. When a
> token's kid is not in the cached JWKS, the JWKS is fetched again.
> *Changes*
> - {{{}TrustedOidcIssuer{}}}, {{RegisterIssuerRequest}} and the admin REST
> resource get the new jwksCa field. Registration validates it and returns 400
> invalid_request when:
> -- it is set but dynamicJwks is false
> -- it is too long
> -- a secret: value has the wrong syntax
> -- inline PEM does not parse
> - {{TRUSTED_OIDC_ISSUERS}} table: new {{jwks_ca}} column ({{{}TEXT{}}}, or
> {{CLOB}} on Oracle).
> - {{TrustedOidcIssuerService}} exposes per-issuer TLS trust. It is used by
> OIDC discovery (Apache HttpClient) and by the JWKS fetch (Nimbus
> {{DefaultResourceRetriever}} with a per-issuer {{{}SSLSocketFactory{}}}).
> - JWTokenAuthority gets an overload of verifyToken that takes the JWKS
> SSLSocketFactory.
> - Docs: new field, config properties and RBAC requirements. Also a
> shared-root option: if every cluster chains to one root CA, use the same
> bundle or Secret for every issuer.
> Acceptance criteria
> - Issuers with different self-signed CAs all validate.
> - Replacing an issuer's CA bundle (inline or Secret) takes effect without a
> Knox restart.
> - An unrelated TLS client in Knox does not trust the issuer CA.
> - Rotating an apiserver signing key needs no Knox config change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
