fmorg-git commented on code in PR #10742: URL: https://github.com/apache/ozone/pull/10742#discussion_r3654991443
########## hadoop-hdds/docs/content/design/common-auth-service.md: ########## @@ -0,0 +1,727 @@ +--- +title: Custos — Protocol-Neutral Pluggable Authentication for Ozone +summary: Custos, a shared service that validates client identity for all Ozone access paths on the server side +date: 2026-07-13 +jira: HDDS-15845 +status: proposed +author: Abhishek Pal +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Custos — Ozone Auth Service Design Doc + +## Table of Contents +1. [Problem Statement](#1-problem-statement) +2. [Scope](#2-scope) +3. [Architecture Overview](#3-architecture-overview) + * [3.1 Request flow](#31-request-flow) + * [3.2 Module structure](#32-module-structure) + * [3.3 Deployment and scaling](#33-deployment-and-scaling) +4. [Background: Existing Authentication Mechanisms](#4-background-existing-authentication-mechanisms) + * [4.1 OzoneFS delegation token path](#41-ozonefs-delegation-token-path) + * [4.2 S3 static-key authentication path](#42-s3-static-key-authentication-path) +5. [Client Use Cases](#5-client-use-cases) + * [5.1 Spark job with many executors (OzoneFS)](#51-spark-job-with-many-executors-ozonefs) + * [5.2 Hive / HiveServer2](#52-hive--hiveserver2) + * [5.3 S3 clients with static keys](#53-s3-clients-with-static-keys) +6. [Custos Provider Interface](#6-custos-provider-interface) +7. [CustosToken and UserInfo](#7-custostoken-and-userinfo) + * [7.1 Token format](#71-token-format) + * [7.2 Token field semantics](#72-token-field-semantics) + * [7.3 Token size](#73-token-size) + * [7.4 UserInfo change and UGI construction](#74-userinfo-change-and-ugi-construction) + * [7.5 OMRequest wire extension](#75-omrequest-wire-extension) +8. [Token Signing and Verification](#8-token-signing-and-verification) + * [8.1 Signing algorithm and key source](#81-signing-algorithm-and-key-source) + * [8.2 Signing input](#82-signing-input) + * [8.3 Verification at OM](#83-verification-at-om) + * [8.4 SCM secret-key integration](#84-scm-secret-key-integration) +9. [Phase 1 — OzoneFS / Hadoop-RPC Clients](#9-phase-1--ozonefs--hadoop-rpc-clients) +10. [Phase 2 — S3 Gateway](#10-phase-2--s3-gateway) +11. [Edge Cases and Security Considerations](#11-edge-cases-and-security-considerations) +12. [Compatibility and Upgrade](#12-compatibility-and-upgrade) +13. [Future Work](#13-future-work) + +--- + +> *Custos* is Latin for "guardian" or "watchman". + +## 1. Problem Statement + +Ozone has several ways for a client to prove who it is. +Each one is written separately, and there is no shared place to plug in a new one. +Three concrete problems follow: + +**(a) Non-Kerberos credentials cannot get a token from OM.** Review Comment: With STS, non-Kerberos credentials will be able to get a token from OM ########## hadoop-hdds/docs/content/design/common-auth-service.md: ########## @@ -0,0 +1,727 @@ +--- +title: Custos — Protocol-Neutral Pluggable Authentication for Ozone +summary: Custos, a shared service that validates client identity for all Ozone access paths on the server side +date: 2026-07-13 +jira: HDDS-15845 +status: proposed +author: Abhishek Pal +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Custos — Ozone Auth Service Design Doc + +## Table of Contents +1. [Problem Statement](#1-problem-statement) +2. [Scope](#2-scope) +3. [Architecture Overview](#3-architecture-overview) + * [3.1 Request flow](#31-request-flow) + * [3.2 Module structure](#32-module-structure) + * [3.3 Deployment and scaling](#33-deployment-and-scaling) +4. [Background: Existing Authentication Mechanisms](#4-background-existing-authentication-mechanisms) + * [4.1 OzoneFS delegation token path](#41-ozonefs-delegation-token-path) + * [4.2 S3 static-key authentication path](#42-s3-static-key-authentication-path) +5. [Client Use Cases](#5-client-use-cases) + * [5.1 Spark job with many executors (OzoneFS)](#51-spark-job-with-many-executors-ozonefs) + * [5.2 Hive / HiveServer2](#52-hive--hiveserver2) + * [5.3 S3 clients with static keys](#53-s3-clients-with-static-keys) +6. [Custos Provider Interface](#6-custos-provider-interface) +7. [CustosToken and UserInfo](#7-custostoken-and-userinfo) + * [7.1 Token format](#71-token-format) + * [7.2 Token field semantics](#72-token-field-semantics) + * [7.3 Token size](#73-token-size) + * [7.4 UserInfo change and UGI construction](#74-userinfo-change-and-ugi-construction) + * [7.5 OMRequest wire extension](#75-omrequest-wire-extension) +8. [Token Signing and Verification](#8-token-signing-and-verification) + * [8.1 Signing algorithm and key source](#81-signing-algorithm-and-key-source) + * [8.2 Signing input](#82-signing-input) + * [8.3 Verification at OM](#83-verification-at-om) + * [8.4 SCM secret-key integration](#84-scm-secret-key-integration) +9. [Phase 1 — OzoneFS / Hadoop-RPC Clients](#9-phase-1--ozonefs--hadoop-rpc-clients) +10. [Phase 2 — S3 Gateway](#10-phase-2--s3-gateway) +11. [Edge Cases and Security Considerations](#11-edge-cases-and-security-considerations) +12. [Compatibility and Upgrade](#12-compatibility-and-upgrade) +13. [Future Work](#13-future-work) + +--- + +> *Custos* is Latin for "guardian" or "watchman". + +## 1. Problem Statement + +Ozone has several ways for a client to prove who it is. +Each one is written separately, and there is no shared place to plug in a new one. +Three concrete problems follow: + +**(a) Non-Kerberos credentials cannot get a token from OM.** +When security is on, OM only issues a delegation token if the RPC connection used Kerberos or a certificate. +This check lives in `OzoneManager.isAllowedDelegationTokenOp()`: + +```java +private boolean isAllowedDelegationTokenOp() throws IOException { + AuthenticationMethod authMethod = getConnectionAuthenticationMethod(); + return !UserGroupInformation.isSecurityEnabled() + || (authMethod == AuthenticationMethod.KERBEROS) + || (authMethod == AuthenticationMethod.KERBEROS_SSL) + || (authMethod == AuthenticationMethod.CERTIFICATE); +} +``` + +A client that authenticated with an OIDC token (from a cloud identity provider) has no Kerberos ticket. +So it cannot obtain a delegation token and cannot use the OzoneFS path at all. Review Comment: If the upcoming STS feature is extended to implement `AssumeRoleWithWebIdentity`, OIDC and JWT can be supported. ########## hadoop-hdds/docs/content/design/common-auth-service.md: ########## @@ -0,0 +1,727 @@ +--- +title: Custos — Protocol-Neutral Pluggable Authentication for Ozone +summary: Custos, a shared service that validates client identity for all Ozone access paths on the server side +date: 2026-07-13 +jira: HDDS-15845 +status: proposed +author: Abhishek Pal +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Custos — Ozone Auth Service Design Doc + +## Table of Contents +1. [Problem Statement](#1-problem-statement) +2. [Scope](#2-scope) +3. [Architecture Overview](#3-architecture-overview) + * [3.1 Request flow](#31-request-flow) + * [3.2 Module structure](#32-module-structure) + * [3.3 Deployment and scaling](#33-deployment-and-scaling) +4. [Background: Existing Authentication Mechanisms](#4-background-existing-authentication-mechanisms) + * [4.1 OzoneFS delegation token path](#41-ozonefs-delegation-token-path) + * [4.2 S3 static-key authentication path](#42-s3-static-key-authentication-path) +5. [Client Use Cases](#5-client-use-cases) + * [5.1 Spark job with many executors (OzoneFS)](#51-spark-job-with-many-executors-ozonefs) + * [5.2 Hive / HiveServer2](#52-hive--hiveserver2) + * [5.3 S3 clients with static keys](#53-s3-clients-with-static-keys) +6. [Custos Provider Interface](#6-custos-provider-interface) +7. [CustosToken and UserInfo](#7-custostoken-and-userinfo) + * [7.1 Token format](#71-token-format) + * [7.2 Token field semantics](#72-token-field-semantics) + * [7.3 Token size](#73-token-size) + * [7.4 UserInfo change and UGI construction](#74-userinfo-change-and-ugi-construction) + * [7.5 OMRequest wire extension](#75-omrequest-wire-extension) +8. [Token Signing and Verification](#8-token-signing-and-verification) + * [8.1 Signing algorithm and key source](#81-signing-algorithm-and-key-source) + * [8.2 Signing input](#82-signing-input) + * [8.3 Verification at OM](#83-verification-at-om) + * [8.4 SCM secret-key integration](#84-scm-secret-key-integration) +9. [Phase 1 — OzoneFS / Hadoop-RPC Clients](#9-phase-1--ozonefs--hadoop-rpc-clients) +10. [Phase 2 — S3 Gateway](#10-phase-2--s3-gateway) +11. [Edge Cases and Security Considerations](#11-edge-cases-and-security-considerations) +12. [Compatibility and Upgrade](#12-compatibility-and-upgrade) +13. [Future Work](#13-future-work) + +--- + +> *Custos* is Latin for "guardian" or "watchman". + +## 1. Problem Statement + +Ozone has several ways for a client to prove who it is. +Each one is written separately, and there is no shared place to plug in a new one. +Three concrete problems follow: + +**(a) Non-Kerberos credentials cannot get a token from OM.** +When security is on, OM only issues a delegation token if the RPC connection used Kerberos or a certificate. +This check lives in `OzoneManager.isAllowedDelegationTokenOp()`: + +```java +private boolean isAllowedDelegationTokenOp() throws IOException { + AuthenticationMethod authMethod = getConnectionAuthenticationMethod(); + return !UserGroupInformation.isSecurityEnabled() + || (authMethod == AuthenticationMethod.KERBEROS) + || (authMethod == AuthenticationMethod.KERBEROS_SSL) + || (authMethod == AuthenticationMethod.CERTIFICATE); +} +``` + +A client that authenticated with an OIDC token (from a cloud identity provider) has no Kerberos ticket. +So it cannot obtain a delegation token and cannot use the OzoneFS path at all. + +**(b) Every credential type is coded on its own, with no shared hook.** +Adding a new credential type today means editing several unrelated places: `OMClientRequest.getUserInfo()`, `OzoneManager.isAllowedDelegationTokenOp()`, and the S3 validation path. +There is no single interface where a new provider can be registered. + +**(c) The identity carried to OM has no group information.** +`OMClientRequest.createUGI()` builds a `UserGroupInformation` from the user name only: + +```java +userGroupInformation = UserGroupInformation.createRemoteUser( + getUserInfo().getUserName()); +``` + +The `UserInfo` protobuf message has no `groups` field. +When an authorization check needs group membership, it must be resolved out of band (for example by Ranger's own group provider). +The token the client presents already knows the groups, but that information is dropped before it reaches OM. + +--- + +## 2. Scope + +Custos is a server-side component that validates client credentials through pluggable providers and issues a signed `CustosToken`. +OM verifies the token locally. +This design covers **authentication only**. +Authorization (policy evaluation) is future work — see [Section 13](#13-future-work). +Custos does **not** replace or skip OM's existing ACL checks; a request verified from a `CustosToken` still runs through the same `checkAcls()` path as any other request. +This keeps authorization exactly where it is today and avoids introducing a second policy decision point in this design. + +Custos is meant to be the single authentication entry point for every Ozone client access path. +This design works two paths through concretely: OzoneFS / Hadoop-RPC clients (Phase 1) and the S3 gateway (Phase 2). +Other access paths follow the same provider model and are not designed here. + +The work is split into two phases. + +**Phase 1 — OzoneFS / Hadoop-RPC clients:** +- Let a driver that authenticated with OIDC (or any configured provider) get a session-scoped `CustosToken` in place of the Kerberos-gated delegation token. +- Keep the executor path unchanged. Executors carry the token in YARN `Credentials` and never contact Custos. +- Keep legacy Kerberos clients working with no change. +- Carry groups inside the token so OM can build a full `UserGroupInformation`. + +**Phase 2 — S3 Gateway:** +- Make Custos the single validation point for S3 credentials, starting with long-lived static keys. +- Route static-key validation through the same provider contract so any other credential type can plug in later without new gateway code. + +**Out of scope:** +- Authorization policy evaluation (Ranger, native ACLs). +- Temporary (session-token) credential validation — see [Section 13](#13-future-work). +- Inter-service authentication (SCM ↔ OM ↔ Datanode). + +--- + +## 3. Architecture Overview + +### 3.1 Request flow + +A client presents a credential to Custos. +Custos validates it through the matching provider and returns a signed `CustosToken`. +The client carries that token to OM on each request. +OM verifies the token locally — no network call — and then runs its existing ACL checks. + +``` +┌────────────┐ ┌───────────────┐ ┌────────────────┐ +│ Client │────▶│ Custos │────▶│ Ozone Manager │ +│ │ │ │ │ │ +│ Presents │ │ 1. Pick │ │ 1. Verify │ +│ credential │ │ provider │ │ token sig │ +│ │ │ 2. Validate │ │ 2. Check │ +│ │ │ credential │ │ audience │ +│ │ │ 3. Issue │ │ + expiry │ +│ │ │ CustosToken│ │ 3. Build UGI │ +│ │ │ │ │ + run ACLs │ +└────────────┘ └───────────────┘ └────────────────┘ +``` + +The token is a bearer of a verified identity, not a proxy for an authorization decision. +OM trusts the *identity* the token asserts (subject and groups) because the signature proves Custos issued it, but OM still evaluates its own ACLs against that identity. + +### 3.2 Module structure + +- **`hadoop-ozone/ozone-custos-common`**: shared library containing the `CustosTokenProto` definition, the `CustosTokenSigner` interface (with the shared `computeSigningInput` used by both signing and verification), and the provider plugin interfaces (`CustosProvider`, `IdentityProvider`, `CustosCredential`, `CustosIdentity`, `CustosException`). Depended on by both Custos and OM. +- **`hadoop-ozone/ozone-custos`**: the standalone Custos service process — the gRPC server, the provider and identity-provider implementations, provider loading, the SCM-backed token signer, token issuance, audit, and metrics. Depends on `hadoop-hdds/framework` for the SCM `SecretKeyClient` (see [Section 8.4](#84-scm-secret-key-integration)). +- **OM** owns its own `CustosTokenVerifier` (in the `ozone-manager` module). OM links only against `ozone-custos-common` for the token type and `computeSigningInput`, and verifies with the SCM `SecretKeyClient` it already holds — so the Custos provider dependencies (for example a JWKS/OIDC client) never enter the OM classpath. + +### 3.3 Deployment and scaling + +Custos is stateless. +It holds no per-client session state: a token is self-describing and is verified from its signature alone. +All signing and verification use the symmetric key that SCM distributes through `SecretKeyClient` / `ManagedSecretKey` ([Section 8.4](#84-scm-secret-key-integration)), so any Custos instance can issue a token that any OM can verify, and OM never has to reach a *specific* Custos instance. + +Because there is no shared state to coordinate, Custos can run as a single instance for a small cluster or as several independent instances behind a load balancer for availability and throughput. +Instance count is independent of the OM count; Custos does not join the OM Ratis ring. + +--- + +## 4. Background: Existing Authentication Mechanisms + +This section describes the authentication paths that exist in Ozone today. +No judgment — just the paths as they are, so the phased changes later are clear. + +### 4.1 OzoneFS delegation token path + +This is how Spark, Hive, and other Hadoop-RPC clients authenticate. + +1. The client calls `FileSystem.getDelegationToken(renewer)`. In Ozone this reaches `BasicOzoneClientAdapterImpl.getDelegationToken()`: + + ```java + public Token<OzoneTokenIdentifier> getDelegationToken(String renewer) + throws IOException { + if (!securityEnabled) { + return null; + } + Token<OzoneTokenIdentifier> token = + ozoneClient.getObjectStore().getDelegationToken(new Text(renewer)); + token.setKind(OzoneTokenIdentifier.KIND_NAME); + return token; + } + ``` + +2. The request reaches OM as `OMGetDelegationTokenRequest`. OM checks `isAllowedDelegationTokenOp()` (shown in [Section 1](#1-problem-statement)) and, if allowed, the `OzoneDelegationTokenSecretManager` signs and returns the token. +3. On every later RPC, the client sends the token. OM validates it with `OzoneDelegationTokenSecretManager.retrievePassword()`. +4. Token renewal is handled by `OzoneDelegationTokenRenewer`, a background thread that YARN runs so the token stays valid even after the driver exits. + +The token identifier kind is `OzoneTokenIdentifier.KIND_NAME` (`"OzoneToken"`). + +There are in fact **two** tokens in this path, and only the first one changes with Custos: + +- `Token<OzoneTokenIdentifier>` — the session-scoped delegation token above. YARN distributes it to every executor and it is used for OM RPCs. +- `Token<OzoneBlockTokenIdentifier>` — a per-block token OM issues during key operations and the client carries to the datanodes for the actual data transfer. + +Custos changes only how the driver bootstraps the first token (Kerberos → OIDC or another provider). +Block tokens, executor distribution, and the datanode data path are untouched. + +### 4.2 S3 static-key authentication path + +This is how S3 clients (boto3, the AWS Java SDK, s5cmd) authenticate with a long-lived access key and secret. + +1. The client signs the request with AWS Signature Version 4 and sends an `Authorization: AWS4-HMAC-SHA256 Credential=<accessId>/...` header. +2. The gateway parses it with `AuthorizationV4HeaderParser` and rebuilds the string-to-sign with `StringToSignProducer`. +3. The gateway forwards the access key, the signature, and the string-to-sign to OM inside the `S3Authentication` part of the request. The gateway does **not** verify the signature itself. +4. OM looks up the secret in `s3SecretTable` and verifies the HMAC with `AWSV4AuthValidator.validateRequest()`. + +So the gateway trusts the client's string-to-sign and OM does the actual HMAC check. +The gateway holds no secret. + +--- + +## 5. Client Use Cases + +Each use case describes the client today and what changes with Custos. + +### 5.1 Spark job with many executors (OzoneFS) + +**Today:** The driver gets a `Token<OzoneTokenIdentifier>` using Kerberos. +YARN distributes it to every executor inside `Credentials`. +Each executor finds the token with `OzoneDelegationTokenSelector` and uses it for every OM RPC. +`OzoneDelegationTokenRenewer` renews it before expiry, even if the driver has exited. +Executors never fetch their own token. + +**With Custos (Phase 1):** The driver authenticates to Custos with its OIDC JWT and receives a session-scoped `CustosToken`. +It is multi-use, has a longer TTL, and carries the driver's groups. Review Comment: I believe you can do this if AssumeRoleWithWebIdentity is implemented in STS ########## hadoop-hdds/docs/content/design/common-auth-service.md: ########## @@ -0,0 +1,727 @@ +--- +title: Custos — Protocol-Neutral Pluggable Authentication for Ozone +summary: Custos, a shared service that validates client identity for all Ozone access paths on the server side +date: 2026-07-13 +jira: HDDS-15845 +status: proposed +author: Abhishek Pal +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Custos — Ozone Auth Service Design Doc + +## Table of Contents +1. [Problem Statement](#1-problem-statement) +2. [Scope](#2-scope) +3. [Architecture Overview](#3-architecture-overview) + * [3.1 Request flow](#31-request-flow) + * [3.2 Module structure](#32-module-structure) + * [3.3 Deployment and scaling](#33-deployment-and-scaling) +4. [Background: Existing Authentication Mechanisms](#4-background-existing-authentication-mechanisms) + * [4.1 OzoneFS delegation token path](#41-ozonefs-delegation-token-path) + * [4.2 S3 static-key authentication path](#42-s3-static-key-authentication-path) +5. [Client Use Cases](#5-client-use-cases) + * [5.1 Spark job with many executors (OzoneFS)](#51-spark-job-with-many-executors-ozonefs) + * [5.2 Hive / HiveServer2](#52-hive--hiveserver2) + * [5.3 S3 clients with static keys](#53-s3-clients-with-static-keys) +6. [Custos Provider Interface](#6-custos-provider-interface) +7. [CustosToken and UserInfo](#7-custostoken-and-userinfo) + * [7.1 Token format](#71-token-format) + * [7.2 Token field semantics](#72-token-field-semantics) + * [7.3 Token size](#73-token-size) + * [7.4 UserInfo change and UGI construction](#74-userinfo-change-and-ugi-construction) + * [7.5 OMRequest wire extension](#75-omrequest-wire-extension) +8. [Token Signing and Verification](#8-token-signing-and-verification) + * [8.1 Signing algorithm and key source](#81-signing-algorithm-and-key-source) + * [8.2 Signing input](#82-signing-input) + * [8.3 Verification at OM](#83-verification-at-om) + * [8.4 SCM secret-key integration](#84-scm-secret-key-integration) +9. [Phase 1 — OzoneFS / Hadoop-RPC Clients](#9-phase-1--ozonefs--hadoop-rpc-clients) +10. [Phase 2 — S3 Gateway](#10-phase-2--s3-gateway) +11. [Edge Cases and Security Considerations](#11-edge-cases-and-security-considerations) +12. [Compatibility and Upgrade](#12-compatibility-and-upgrade) +13. [Future Work](#13-future-work) + +--- + +> *Custos* is Latin for "guardian" or "watchman". + +## 1. Problem Statement + +Ozone has several ways for a client to prove who it is. +Each one is written separately, and there is no shared place to plug in a new one. +Three concrete problems follow: + +**(a) Non-Kerberos credentials cannot get a token from OM.** +When security is on, OM only issues a delegation token if the RPC connection used Kerberos or a certificate. +This check lives in `OzoneManager.isAllowedDelegationTokenOp()`: + +```java +private boolean isAllowedDelegationTokenOp() throws IOException { + AuthenticationMethod authMethod = getConnectionAuthenticationMethod(); + return !UserGroupInformation.isSecurityEnabled() + || (authMethod == AuthenticationMethod.KERBEROS) + || (authMethod == AuthenticationMethod.KERBEROS_SSL) + || (authMethod == AuthenticationMethod.CERTIFICATE); +} +``` + +A client that authenticated with an OIDC token (from a cloud identity provider) has no Kerberos ticket. +So it cannot obtain a delegation token and cannot use the OzoneFS path at all. + +**(b) Every credential type is coded on its own, with no shared hook.** +Adding a new credential type today means editing several unrelated places: `OMClientRequest.getUserInfo()`, `OzoneManager.isAllowedDelegationTokenOp()`, and the S3 validation path. +There is no single interface where a new provider can be registered. + +**(c) The identity carried to OM has no group information.** +`OMClientRequest.createUGI()` builds a `UserGroupInformation` from the user name only: + +```java +userGroupInformation = UserGroupInformation.createRemoteUser( + getUserInfo().getUserName()); +``` + +The `UserInfo` protobuf message has no `groups` field. +When an authorization check needs group membership, it must be resolved out of band (for example by Ranger's own group provider). +The token the client presents already knows the groups, but that information is dropped before it reaches OM. + +--- + +## 2. Scope + +Custos is a server-side component that validates client credentials through pluggable providers and issues a signed `CustosToken`. +OM verifies the token locally. +This design covers **authentication only**. +Authorization (policy evaluation) is future work — see [Section 13](#13-future-work). +Custos does **not** replace or skip OM's existing ACL checks; a request verified from a `CustosToken` still runs through the same `checkAcls()` path as any other request. +This keeps authorization exactly where it is today and avoids introducing a second policy decision point in this design. + +Custos is meant to be the single authentication entry point for every Ozone client access path. +This design works two paths through concretely: OzoneFS / Hadoop-RPC clients (Phase 1) and the S3 gateway (Phase 2). +Other access paths follow the same provider model and are not designed here. + +The work is split into two phases. + +**Phase 1 — OzoneFS / Hadoop-RPC clients:** +- Let a driver that authenticated with OIDC (or any configured provider) get a session-scoped `CustosToken` in place of the Kerberos-gated delegation token. +- Keep the executor path unchanged. Executors carry the token in YARN `Credentials` and never contact Custos. +- Keep legacy Kerberos clients working with no change. +- Carry groups inside the token so OM can build a full `UserGroupInformation`. + +**Phase 2 — S3 Gateway:** +- Make Custos the single validation point for S3 credentials, starting with long-lived static keys. +- Route static-key validation through the same provider contract so any other credential type can plug in later without new gateway code. + +**Out of scope:** +- Authorization policy evaluation (Ranger, native ACLs). +- Temporary (session-token) credential validation — see [Section 13](#13-future-work). +- Inter-service authentication (SCM ↔ OM ↔ Datanode). + +--- + +## 3. Architecture Overview + +### 3.1 Request flow + +A client presents a credential to Custos. +Custos validates it through the matching provider and returns a signed `CustosToken`. +The client carries that token to OM on each request. +OM verifies the token locally — no network call — and then runs its existing ACL checks. + +``` +┌────────────┐ ┌───────────────┐ ┌────────────────┐ +│ Client │────▶│ Custos │────▶│ Ozone Manager │ +│ │ │ │ │ │ +│ Presents │ │ 1. Pick │ │ 1. Verify │ +│ credential │ │ provider │ │ token sig │ +│ │ │ 2. Validate │ │ 2. Check │ +│ │ │ credential │ │ audience │ +│ │ │ 3. Issue │ │ + expiry │ +│ │ │ CustosToken│ │ 3. Build UGI │ +│ │ │ │ │ + run ACLs │ +└────────────┘ └───────────────┘ └────────────────┘ +``` + +The token is a bearer of a verified identity, not a proxy for an authorization decision. +OM trusts the *identity* the token asserts (subject and groups) because the signature proves Custos issued it, but OM still evaluates its own ACLs against that identity. + +### 3.2 Module structure + +- **`hadoop-ozone/ozone-custos-common`**: shared library containing the `CustosTokenProto` definition, the `CustosTokenSigner` interface (with the shared `computeSigningInput` used by both signing and verification), and the provider plugin interfaces (`CustosProvider`, `IdentityProvider`, `CustosCredential`, `CustosIdentity`, `CustosException`). Depended on by both Custos and OM. +- **`hadoop-ozone/ozone-custos`**: the standalone Custos service process — the gRPC server, the provider and identity-provider implementations, provider loading, the SCM-backed token signer, token issuance, audit, and metrics. Depends on `hadoop-hdds/framework` for the SCM `SecretKeyClient` (see [Section 8.4](#84-scm-secret-key-integration)). +- **OM** owns its own `CustosTokenVerifier` (in the `ozone-manager` module). OM links only against `ozone-custos-common` for the token type and `computeSigningInput`, and verifies with the SCM `SecretKeyClient` it already holds — so the Custos provider dependencies (for example a JWKS/OIDC client) never enter the OM classpath. + +### 3.3 Deployment and scaling + +Custos is stateless. +It holds no per-client session state: a token is self-describing and is verified from its signature alone. +All signing and verification use the symmetric key that SCM distributes through `SecretKeyClient` / `ManagedSecretKey` ([Section 8.4](#84-scm-secret-key-integration)), so any Custos instance can issue a token that any OM can verify, and OM never has to reach a *specific* Custos instance. + +Because there is no shared state to coordinate, Custos can run as a single instance for a small cluster or as several independent instances behind a load balancer for availability and throughput. +Instance count is independent of the OM count; Custos does not join the OM Ratis ring. + +--- + +## 4. Background: Existing Authentication Mechanisms + +This section describes the authentication paths that exist in Ozone today. +No judgment — just the paths as they are, so the phased changes later are clear. + +### 4.1 OzoneFS delegation token path + +This is how Spark, Hive, and other Hadoop-RPC clients authenticate. + +1. The client calls `FileSystem.getDelegationToken(renewer)`. In Ozone this reaches `BasicOzoneClientAdapterImpl.getDelegationToken()`: + + ```java + public Token<OzoneTokenIdentifier> getDelegationToken(String renewer) + throws IOException { + if (!securityEnabled) { + return null; + } + Token<OzoneTokenIdentifier> token = + ozoneClient.getObjectStore().getDelegationToken(new Text(renewer)); + token.setKind(OzoneTokenIdentifier.KIND_NAME); + return token; + } + ``` + +2. The request reaches OM as `OMGetDelegationTokenRequest`. OM checks `isAllowedDelegationTokenOp()` (shown in [Section 1](#1-problem-statement)) and, if allowed, the `OzoneDelegationTokenSecretManager` signs and returns the token. +3. On every later RPC, the client sends the token. OM validates it with `OzoneDelegationTokenSecretManager.retrievePassword()`. +4. Token renewal is handled by `OzoneDelegationTokenRenewer`, a background thread that YARN runs so the token stays valid even after the driver exits. + +The token identifier kind is `OzoneTokenIdentifier.KIND_NAME` (`"OzoneToken"`). + +There are in fact **two** tokens in this path, and only the first one changes with Custos: + +- `Token<OzoneTokenIdentifier>` — the session-scoped delegation token above. YARN distributes it to every executor and it is used for OM RPCs. +- `Token<OzoneBlockTokenIdentifier>` — a per-block token OM issues during key operations and the client carries to the datanodes for the actual data transfer. + +Custos changes only how the driver bootstraps the first token (Kerberos → OIDC or another provider). +Block tokens, executor distribution, and the datanode data path are untouched. + +### 4.2 S3 static-key authentication path + +This is how S3 clients (boto3, the AWS Java SDK, s5cmd) authenticate with a long-lived access key and secret. + +1. The client signs the request with AWS Signature Version 4 and sends an `Authorization: AWS4-HMAC-SHA256 Credential=<accessId>/...` header. +2. The gateway parses it with `AuthorizationV4HeaderParser` and rebuilds the string-to-sign with `StringToSignProducer`. +3. The gateway forwards the access key, the signature, and the string-to-sign to OM inside the `S3Authentication` part of the request. The gateway does **not** verify the signature itself. +4. OM looks up the secret in `s3SecretTable` and verifies the HMAC with `AWSV4AuthValidator.validateRequest()`. + +So the gateway trusts the client's string-to-sign and OM does the actual HMAC check. +The gateway holds no secret. + +--- + +## 5. Client Use Cases + +Each use case describes the client today and what changes with Custos. + +### 5.1 Spark job with many executors (OzoneFS) + +**Today:** The driver gets a `Token<OzoneTokenIdentifier>` using Kerberos. +YARN distributes it to every executor inside `Credentials`. +Each executor finds the token with `OzoneDelegationTokenSelector` and uses it for every OM RPC. +`OzoneDelegationTokenRenewer` renews it before expiry, even if the driver has exited. +Executors never fetch their own token. + +**With Custos (Phase 1):** The driver authenticates to Custos with its OIDC JWT and receives a session-scoped `CustosToken`. +It is multi-use, has a longer TTL, and carries the driver's groups. +The token is session-scoped rather than per-operation on purpose: a single filesystem operation already spans several OM RPCs (open, get key info, commit, and so on), and a job issues many such operations, so a per-operation token would force a Custos round trip on the hot path. +A session-scoped multi-use token matches how the delegation token works today, and how AWS STS session credentials work. +YARN distributes it to executors exactly as before — it is still a `Token<>` inside `Credentials`. +Executors put it in `OMRequest.custosToken` (field 154, the next free `OMRequest` field number). +OM verifies the signature locally. +No executor contacts Custos. +The renewer calls the Custos renew endpoint instead of the delegation token renew path. + +### 5.2 Hive / HiveServer2 + +**Today:** HiveServer2 is a long-running daemon. +At startup it calls `fs.addDelegationTokens()` and holds one token for the cluster lifetime, renewing it through `OzoneDelegationTokenRenewer`. +This requires Kerberos. + +**With Custos:** HiveServer2 authenticates once with OIDC and gets a token with a bounded `maxLifetimeMs` (for example 7 days). +It renews daily through the Custos renew endpoint until the maximum lifetime is reached, after which it must re-authenticate. + +### 5.3 S3 clients with static keys + +**Today:** boto3 signs with SigV4. +The gateway builds the string-to-sign and forwards it to OM, which verifies the HMAC against `s3SecretTable`. +This works. + +**With Custos (Phase 2):** The verification moves into Custos (through `S3SecretProvider`, see [Section 6](#6-custos-provider-interface)), which then issues a `CustosToken`. +The client sees no difference — the same keys and the same requests keep working. + +--- + +## 6. Custos Provider Interface + +The pluggable provider model is the core of Custos, split into two contracts so that credential *validation* and identity *enrichment* can vary independently. + +**Authentication — `CustosProvider`.** +A `CustosProvider` validates one credential type and returns the authenticated subject (a principal name). Custos selects the provider from the credential type. + +```java +public interface CustosProvider { + String validateSubject(CustosCredential credential) + throws CustosException; +} +``` + +**Identity resolution — `IdentityProvider`.** +Once a subject is authenticated, an `IdentityProvider` resolves the full `CustosIdentity` for it — the groups and roles that go into the token. Keeping this separate lets, for example, a SPNEGO credential be validated by `KerberosProvider` while groups are resolved by `KerberosIdentityProvider` (Hadoop group mapping), or an OIDC credential be validated once and enriched from a directory. + +- `CustosCredential` is the single input to `CustosProvider`. It carries the credential type and the raw credential material — a bearer token, a SPNEGO token, or S3 SigV4 material — so all providers share one input and Custos can route on type without special-casing. +- `CustosIdentity` is the result of identity resolution. Its fields are `subject`, `groups`, `roles`, `issuer`, `authMethod`, `authenticatedAt`, and `expiresAt`. + +Custos ships these providers (authentication) and identity providers (resolution): + +| Provider | Credential / role | Behavior | +|:---------|:------------------|:---------| +| `KerberosProvider` (auth) | SPNEGO token | Accept the client SPNEGO token with the Custos service keytab (`ozone.custos.kerberos.keytab`). The service principal travels in the credential from the client. | +| `OidcProvider` (auth) | OIDC JWT (bearer token) | Validate the JWT against a JWKS endpoint (RSA keys only, reject `alg=none`); the subject and claims come from the token. | Review Comment: Having the service keytab that creates tokens and having the service principal travel in the credential is precisely how the STS design works. ########## hadoop-hdds/docs/content/design/common-auth-service.md: ########## @@ -0,0 +1,727 @@ +--- +title: Custos — Protocol-Neutral Pluggable Authentication for Ozone +summary: Custos, a shared service that validates client identity for all Ozone access paths on the server side +date: 2026-07-13 +jira: HDDS-15845 +status: proposed +author: Abhishek Pal +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Custos — Ozone Auth Service Design Doc + +## Table of Contents +1. [Problem Statement](#1-problem-statement) +2. [Scope](#2-scope) +3. [Architecture Overview](#3-architecture-overview) + * [3.1 Request flow](#31-request-flow) + * [3.2 Module structure](#32-module-structure) + * [3.3 Deployment and scaling](#33-deployment-and-scaling) +4. [Background: Existing Authentication Mechanisms](#4-background-existing-authentication-mechanisms) + * [4.1 OzoneFS delegation token path](#41-ozonefs-delegation-token-path) + * [4.2 S3 static-key authentication path](#42-s3-static-key-authentication-path) +5. [Client Use Cases](#5-client-use-cases) + * [5.1 Spark job with many executors (OzoneFS)](#51-spark-job-with-many-executors-ozonefs) + * [5.2 Hive / HiveServer2](#52-hive--hiveserver2) + * [5.3 S3 clients with static keys](#53-s3-clients-with-static-keys) +6. [Custos Provider Interface](#6-custos-provider-interface) +7. [CustosToken and UserInfo](#7-custostoken-and-userinfo) + * [7.1 Token format](#71-token-format) + * [7.2 Token field semantics](#72-token-field-semantics) + * [7.3 Token size](#73-token-size) + * [7.4 UserInfo change and UGI construction](#74-userinfo-change-and-ugi-construction) + * [7.5 OMRequest wire extension](#75-omrequest-wire-extension) +8. [Token Signing and Verification](#8-token-signing-and-verification) + * [8.1 Signing algorithm and key source](#81-signing-algorithm-and-key-source) + * [8.2 Signing input](#82-signing-input) + * [8.3 Verification at OM](#83-verification-at-om) + * [8.4 SCM secret-key integration](#84-scm-secret-key-integration) +9. [Phase 1 — OzoneFS / Hadoop-RPC Clients](#9-phase-1--ozonefs--hadoop-rpc-clients) +10. [Phase 2 — S3 Gateway](#10-phase-2--s3-gateway) +11. [Edge Cases and Security Considerations](#11-edge-cases-and-security-considerations) +12. [Compatibility and Upgrade](#12-compatibility-and-upgrade) +13. [Future Work](#13-future-work) + +--- + +> *Custos* is Latin for "guardian" or "watchman". + +## 1. Problem Statement + +Ozone has several ways for a client to prove who it is. +Each one is written separately, and there is no shared place to plug in a new one. +Three concrete problems follow: + +**(a) Non-Kerberos credentials cannot get a token from OM.** +When security is on, OM only issues a delegation token if the RPC connection used Kerberos or a certificate. +This check lives in `OzoneManager.isAllowedDelegationTokenOp()`: + +```java +private boolean isAllowedDelegationTokenOp() throws IOException { + AuthenticationMethod authMethod = getConnectionAuthenticationMethod(); + return !UserGroupInformation.isSecurityEnabled() + || (authMethod == AuthenticationMethod.KERBEROS) + || (authMethod == AuthenticationMethod.KERBEROS_SSL) + || (authMethod == AuthenticationMethod.CERTIFICATE); +} +``` + +A client that authenticated with an OIDC token (from a cloud identity provider) has no Kerberos ticket. +So it cannot obtain a delegation token and cannot use the OzoneFS path at all. + +**(b) Every credential type is coded on its own, with no shared hook.** +Adding a new credential type today means editing several unrelated places: `OMClientRequest.getUserInfo()`, `OzoneManager.isAllowedDelegationTokenOp()`, and the S3 validation path. +There is no single interface where a new provider can be registered. + +**(c) The identity carried to OM has no group information.** +`OMClientRequest.createUGI()` builds a `UserGroupInformation` from the user name only: + +```java +userGroupInformation = UserGroupInformation.createRemoteUser( + getUserInfo().getUserName()); +``` + +The `UserInfo` protobuf message has no `groups` field. +When an authorization check needs group membership, it must be resolved out of band (for example by Ranger's own group provider). +The token the client presents already knows the groups, but that information is dropped before it reaches OM. + +--- + +## 2. Scope + +Custos is a server-side component that validates client credentials through pluggable providers and issues a signed `CustosToken`. +OM verifies the token locally. +This design covers **authentication only**. +Authorization (policy evaluation) is future work — see [Section 13](#13-future-work). +Custos does **not** replace or skip OM's existing ACL checks; a request verified from a `CustosToken` still runs through the same `checkAcls()` path as any other request. +This keeps authorization exactly where it is today and avoids introducing a second policy decision point in this design. + +Custos is meant to be the single authentication entry point for every Ozone client access path. +This design works two paths through concretely: OzoneFS / Hadoop-RPC clients (Phase 1) and the S3 gateway (Phase 2). +Other access paths follow the same provider model and are not designed here. + +The work is split into two phases. + +**Phase 1 — OzoneFS / Hadoop-RPC clients:** +- Let a driver that authenticated with OIDC (or any configured provider) get a session-scoped `CustosToken` in place of the Kerberos-gated delegation token. +- Keep the executor path unchanged. Executors carry the token in YARN `Credentials` and never contact Custos. +- Keep legacy Kerberos clients working with no change. +- Carry groups inside the token so OM can build a full `UserGroupInformation`. + +**Phase 2 — S3 Gateway:** +- Make Custos the single validation point for S3 credentials, starting with long-lived static keys. +- Route static-key validation through the same provider contract so any other credential type can plug in later without new gateway code. + +**Out of scope:** +- Authorization policy evaluation (Ranger, native ACLs). +- Temporary (session-token) credential validation — see [Section 13](#13-future-work). +- Inter-service authentication (SCM ↔ OM ↔ Datanode). + +--- + +## 3. Architecture Overview + +### 3.1 Request flow + +A client presents a credential to Custos. +Custos validates it through the matching provider and returns a signed `CustosToken`. +The client carries that token to OM on each request. +OM verifies the token locally — no network call — and then runs its existing ACL checks. + +``` +┌────────────┐ ┌───────────────┐ ┌────────────────┐ +│ Client │────▶│ Custos │────▶│ Ozone Manager │ +│ │ │ │ │ │ +│ Presents │ │ 1. Pick │ │ 1. Verify │ +│ credential │ │ provider │ │ token sig │ +│ │ │ 2. Validate │ │ 2. Check │ +│ │ │ credential │ │ audience │ +│ │ │ 3. Issue │ │ + expiry │ +│ │ │ CustosToken│ │ 3. Build UGI │ +│ │ │ │ │ + run ACLs │ +└────────────┘ └───────────────┘ └────────────────┘ +``` + +The token is a bearer of a verified identity, not a proxy for an authorization decision. +OM trusts the *identity* the token asserts (subject and groups) because the signature proves Custos issued it, but OM still evaluates its own ACLs against that identity. + +### 3.2 Module structure + +- **`hadoop-ozone/ozone-custos-common`**: shared library containing the `CustosTokenProto` definition, the `CustosTokenSigner` interface (with the shared `computeSigningInput` used by both signing and verification), and the provider plugin interfaces (`CustosProvider`, `IdentityProvider`, `CustosCredential`, `CustosIdentity`, `CustosException`). Depended on by both Custos and OM. +- **`hadoop-ozone/ozone-custos`**: the standalone Custos service process — the gRPC server, the provider and identity-provider implementations, provider loading, the SCM-backed token signer, token issuance, audit, and metrics. Depends on `hadoop-hdds/framework` for the SCM `SecretKeyClient` (see [Section 8.4](#84-scm-secret-key-integration)). +- **OM** owns its own `CustosTokenVerifier` (in the `ozone-manager` module). OM links only against `ozone-custos-common` for the token type and `computeSigningInput`, and verifies with the SCM `SecretKeyClient` it already holds — so the Custos provider dependencies (for example a JWKS/OIDC client) never enter the OM classpath. + +### 3.3 Deployment and scaling + +Custos is stateless. +It holds no per-client session state: a token is self-describing and is verified from its signature alone. +All signing and verification use the symmetric key that SCM distributes through `SecretKeyClient` / `ManagedSecretKey` ([Section 8.4](#84-scm-secret-key-integration)), so any Custos instance can issue a token that any OM can verify, and OM never has to reach a *specific* Custos instance. + Review Comment: It appears that this design `borrows heavily` from the upcoming STS feature (using symmetric ManagedSecretKey for verification, verifying the token then checking ACL calls, including the signature and signingKeyId in the token itself, instead of STSTokenVerifier there is CustosTokenVerifier, have session-scoped tokens with expiration, etc). Would it be more reasonable to leverage the upcoming STS work and extend it for OIDC/JWT instead of implementing something else that is substantially similar? @ChenSammi @errose28 @umamaheswararao Thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
