Just to reiterate my points discussed in the community sync here: the more I think about it the more I agree the OAuth endpoint *should be removed from the REST spec*. Even though the endpoint is optional, and even if we do not care about the security concerns, it still provides users an impression that the endpoint "should" be implemented, or "is the preferred authentication mechanism". And as we have found out, the server capability proposal does not cover this case since this is the first endpoint to hit before the GetConfig endpoint.
As Ryan said, if we want to do that we need an alternative plan. I don't have anything concrete, but here is my line of thought: 1. remove OAuth2 endpoint from the "REST OpenAPI spec" 2. create a client-side interface (in each language) that different authentication mechanisms can be plugged in to talk to the REST catalog 3. refactor and make OAuth2 an implementation of that interface. I can also help with doing the same for AWS Sigv4, and the community can further support some additional ones like Kerberos, SAML, Google SSO, etc. based on the individual use cases. 4. turn 2 + 3 into a "REST catalog authentication spec" that documents about all the supported authentication mechanisms and their defaults. For OAuth2, the default is to have the auth server at the same endpoint as the resource server for backwards compatibility, but that is a configurable property, and we could recommend not to do that based on security concerns. Best, Jack Ye On Wed, May 29, 2024 at 10:28 AM Steven Wu <stevenz...@gmail.com> wrote: > Wondering if the auth endpoints can be separated out to a separate OpenAPI > spec file. Then we still have some reference for interactions with auth > server and make it clear it is not required as part of the REST catalog > server. In most enterprise environments, auth server is likely a separate > server. > > On Tue, May 28, 2024 at 1:25 PM Alex Dutra <alex.du...@dremio.com.invalid> > wrote: > >> Hi, >> >> >>> On point 4, isn't that possible today, Can't that be achieved with the >>> current token exchange approach, and the internal implementation of the >>> endpoint? >> >> >> Unfortunately, no. Token exchange is not widely adopted yet: for example, >> Keycloak has only partial support for it, and Authelia, or Authentik, have >> no support for it at all. >> >> This, and a few other technical issues with the current internals of the >> REST client, makes it nearly impossible to achieve a good integration of >> Iceberg REST with the majority of popular OSS authorization servers. >> >> I am planning to start another email thread to discuss these >> practicalities, but let's first reach consensus on the broader security >> issues voiced here, before we tackle the details. >> >> Thanks, >> >> Alex Dutra >> >> On Tue, May 28, 2024 at 8:41 PM Amogh Jahagirdar <am...@tabular.io> >> wrote: >> >>> I disagree with removing "/v1/oauth/tokens" and I think I also disagree >>> with the premise that implementing that endpoint is required, but I can >>> understand how that's not clear in the spec. I think we can address the >>> required vs non-required discussion with the capabilities PR. >>> <https://github.com/apache/iceberg/pull/9940> >>> >>> It seems like another part of what's driving this discussion is some >>> concern around how do we enforce REST catalog implementations which do >>> implement this endpoint to make sure that the implementation is secure (for >>> example to avoid the MITM example that was brought up). This is ultimately >>> a runtime detail. To me it seems like if we make it clear that such an >>> endpoint should be implemented respecting OAuth2 standards, and we know >>> that OAuth2 compliance requires avoiding that MITM situation, then runtime >>> implementations should just follow the spec there >>> >>> >3. Enable flexibility for Iceberg REST servers to opt for other >>> authorization mechanisms than OAuth 2.0. >>> >4. Enable REST servers to opt for integrating with any standard OAuth2 >>> / >>> OIDC provider (e.g. Okta, Keycloak, Authelia). >>> >>> I agree with both of these points; again I don't think the intention is >>> Oauth2 is the only way, but I think the capabilities PR will make that even >>> more clear. >>> On point 4, isn't that possible today, Can't that be achieved with the >>> current token exchange approach, and the internal implementation of the >>> endpoint? Sorry if I missed that explanation. >>> >>> Thanks, >>> >>> Amogh Jahagirdar >>> >>> On Tue, May 28, 2024 at 11:13 AM Yufei Gu <flyrain...@gmail.com> wrote: >>> >>>> Not an expert on authentication, but reading from the context, I agree >>>> that it’s not a good practice to use a resource server as a token server. >>>> The resource server would need to securely handle and store credentials or >>>> tokens, increasing the risk of credential theft or leakage. Making the >>>> token endpoint optional will mitigate the issue a bit. But if we want to >>>> disable it completely, it's better to do it now to prevent any issues and >>>> migration costs in the future. Can we have a consensus on it? >>>> >>>> >>>> I would prefer to deprecate it to prevent any intentional and >>>> unintentional misuse. We will also need to change the clients since it >>>> connects to the endpoint by default. >>>> >>>> >>>> Yufei >>>> >>>> >>>> On Tue, May 28, 2024 at 8:27 AM Jack Ye <yezhao...@gmail.com> wrote: >>>> >>>>> Sounds like we should try to finalize a consensus around >>>>> https://github.com/apache/iceberg/pull/9940, so that we make it very >>>>> clear what APIs/features are optional. >>>>> >>>>> -Jack >>>>> >>>>> On Tue, May 28, 2024 at 7:25 AM Fokko Driesprong <fo...@apache.org> >>>>> wrote: >>>>> >>>>>> Hey Robert, >>>>>> >>>>>> Sorry for the late reply as I was out last week. I'm not an OAuth >>>>>> guru either, but some context from my end. >>>>>> >>>>>> * Credentials (for example username/password) must _never_ be sent to >>>>>>> the resource server, only to the authorization server. >>>>>> >>>>>> >>>>>> In an earlier discussion >>>>>> <https://github.com/apache/iceberg/pull/8976>, it was agreed that >>>>>> the resource server can also function as the authorization server. But >>>>>> the >>>>>> roles can also be separate. >>>>>> >>>>>> 1.2. As long as OAuth2 is the only mechanism supported by the Iceberg >>>>>>> client, make the existing client parameter “oauth2-server-uri” >>>>>>> mandatory. The Iceberg REST catalog must fail to initialize if the >>>>>>> “oauth2-server-uri” parameter is not defined. >>>>>> >>>>>> >>>>>> It can also be that there is no authentication in the case of an >>>>>> internal REST catalog. For example, the iceberg-rest-image >>>>>> <https://github.com/tabular-io/iceberg-rest-image> that we use for >>>>>> integration tests in PyIceberg. >>>>>> >>>>>> We think that Apache Iceberg REST Catalog spec should not mandate >>>>>>> that a >>>>>>> catalog implementation responds to requests to produce Auth Tokens >>>>>>> (since the REST spec v1 defines a /v1/tokens endpoint, current >>>>>>> implementations have to take deliberate actions when responding to >>>>>>> those >>>>>>> requests, whether with successful token responses or with “access >>>>>>> denied” or “unsupported” responses). >>>>>> >>>>>> The `/v1/tokens` endpoint is optional >>>>>> <https://github.com/apache/iceberg-python/blob/756ae625a2ea0f9c12df78430512ce991f6a1976/pyiceberg/catalog/rest.py#L488-L489> >>>>>> . >>>>>> >>>>>> * Credentials (for example username/password) must _never_ be sent to >>>>>>> the resource server, only to the authorization server. >>>>>> >>>>>> >>>>>> I fully agree! >>>>>> >>>>>> Even if an Iceberg REST server does not implement the >>>>>>> ‘/v1/oauth/tokens’ >>>>>>> endpoint, it can still receive requests to ‘/v1/oauth/tokens’ >>>>>>> containing >>>>>>> clear text credentials, if clients are misconfigured (humans do make >>>>>>> mistakes) - it’s a non-zero risk - bad actors can implement/intercept >>>>>>> that ‘/v1/oauth/tokens’ endpoint and just wait for misconfigured >>>>>>> clients to send credentials. >>>>>> >>>>>> >>>>>> I think the wording is chosen badly. It should not send >>>>>> any credentials, but the code (as in this example >>>>>> <https://developers.google.com/identity/protocols/oauth2#installed> by >>>>>> GCS). >>>>>> >>>>>> I think Jack makes a good point with AWS SigV4 Authentication. I >>>>>>> suppose, in REST Catalog implementations that support that auth method, >>>>>>> the >>>>>>> /v1/oauth/token Catalog REST endpoint is redundant. >>>>>>> >>>>>> >>>>>> There are other cloud providers next to AWS. >>>>>> >>>>>> Kind regards, >>>>>> Fokko >>>>>> >>>>>> >>>>>> >>>>>> Op do 23 mei 2024 om 15:49 schreef Dmitri Bourlatchkov >>>>>> <dmitri.bourlatch...@dremio.com.invalid>: >>>>>> >>>>>>> I think Jack makes a good point with AWS SigV4 Authentication. I >>>>>>> suppose, in REST Catalog implementations that support that auth method, >>>>>>> the >>>>>>> /v1/oauth/token Catalog REST endpoint is redundant. >>>>>>> >>>>>>> Cheers, >>>>>>> Dmitri. >>>>>>> >>>>>>> On Thu, May 23, 2024 at 9:20 AM Jack Ye <yezhao...@gmail.com> wrote: >>>>>>> >>>>>>>> I do not know enough details about OAuth to make comments about >>>>>>>> this issue, but just regarding the statement "OAuth2 is the only >>>>>>>> mechanism >>>>>>>> supported by the Iceberg client", AWS Sigv4 auth is also supported, at >>>>>>>> least in the Java client implementation >>>>>>>> <https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java#L72>. >>>>>>>> It would be nice if we formalize that in the spec, at least define it >>>>>>>> as a >>>>>>>> generic authorization header. >>>>>>>> >>>>>>>> Best, >>>>>>>> Jack Ye >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, May 23, 2024 at 2:51 AM Robert Stupp <sn...@snazy.de> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Iceberg REST implementations, either accessible on the public >>>>>>>>> internet >>>>>>>>> or inside an organization, are usually being secured using >>>>>>>>> appropriate >>>>>>>>> authorization mechanisms. The Nessie team is looking at >>>>>>>>> implementing the >>>>>>>>> Iceberg REST specification and have some questions around the >>>>>>>>> security >>>>>>>>> endpoint(s) defined in the spec. >>>>>>>>> >>>>>>>>> TL;DR we have questions (potentially concerns) about having the >>>>>>>>> ‘/v1/oauth/tokens’ endpoint, for the reasons explained below. We >>>>>>>>> think >>>>>>>>> that ‘/v1/oauth/tokens’ poses potential security and OAuth2 >>>>>>>>> compliance >>>>>>>>> issues, and imposes how authorization should be implemented. >>>>>>>>> * As an open table format, it would be good for Iceberg to focus >>>>>>>>> on the >>>>>>>>> table format / catalog and not how authorization is implemented. >>>>>>>>> The >>>>>>>>> existence of an OAuth endpoint pushes implementations to adopt >>>>>>>>> authorization using only OAuth, whereas the implementers might >>>>>>>>> choose >>>>>>>>> several other ways to implement authorization (e.g. SAML). In our >>>>>>>>> opinion the spec should leave it open to the implementation to >>>>>>>>> decide >>>>>>>>> how authorization will be implemented. >>>>>>>>> * The existence of that endpoint also pushes operators of Iceberg >>>>>>>>> REST >>>>>>>>> endpoints into the authorization service business. >>>>>>>>> * Clients might expose their clear-text credentials to the wrong >>>>>>>>> service, if the (correct) OAuth endpoint is not configured (humans >>>>>>>>> do >>>>>>>>> make mistakes). >>>>>>>>> * (Naive) Iceberg REST servers may proxy requests received for >>>>>>>>> ‘/v1/oauth/tokens’ - and effectively become a “man-in-the-middle”, >>>>>>>>> which >>>>>>>>> is not fully compliant with the OAuth 2.0 specification. >>>>>>>>> >>>>>>>>> Our goals with this discussion are: >>>>>>>>> 1. Secure the Iceberg REST specification by preventing accidental >>>>>>>>> misuse/misimplementation. >>>>>>>>> 2. Prevent that Iceberg REST to get into dictating the >>>>>>>>> “authorization >>>>>>>>> server specifics”. >>>>>>>>> 3. Enable flexibility for Iceberg REST servers to opt for other >>>>>>>>> authorization mechanisms than OAuth 2.0. >>>>>>>>> 4. Enable REST servers to opt for integrating with any standard >>>>>>>>> OAuth2 / >>>>>>>>> OIDC provider (e.g. Okta, Keycloak, Authelia). >>>>>>>>> >>>>>>>>> OAuth 2.0 [1] is one of the common standards accepted in the >>>>>>>>> industry. >>>>>>>>> It defines a secure mechanism to access resources (here: Iceberg >>>>>>>>> REST >>>>>>>>> endpoints). The most important aspect for OAuth 2.0 resources is >>>>>>>>> that >>>>>>>>> (Iceberg REST) servers do not (have to) support password >>>>>>>>> authentication, >>>>>>>>> especially considering the security weaknesses inherent in >>>>>>>>> passwords. >>>>>>>>> Compromised passwords result in compromised data protected by that >>>>>>>>> password. >>>>>>>>> >>>>>>>>> Therefore OAuth 2.0 defines a set of strict rules. Some of these >>>>>>>>> are: >>>>>>>>> * Credentials (for example username/password) must _never_ be sent >>>>>>>>> to >>>>>>>>> the resource server, only to the authorization server. >>>>>>>>> * OAuth 2.0 refresh tokens must _never_ be sent to the resource >>>>>>>>> server, >>>>>>>>> only to the authorization server. (“Unlike access tokens, refresh >>>>>>>>> tokens >>>>>>>>> are intended for use only with authorization servers and are never >>>>>>>>> sent >>>>>>>>> to resource servers.”, cite from section 1.5 of the OAuth RFC >>>>>>>>> 6749.) >>>>>>>>> * While the OAuth RFC states "The authorization server may be the >>>>>>>>> same >>>>>>>>> server as the resource server or a separate entity", this should >>>>>>>>> not be >>>>>>>>> mandated. i.e the spec should be open to supporting >>>>>>>>> implementations that >>>>>>>>> have the authorization server and resource server co-located as >>>>>>>>> well as >>>>>>>>> separate. >>>>>>>>> >>>>>>>>> The Iceberg PR 4771 [2] added the OpenAPI path ‘/v1/oauth/tokens’, >>>>>>>>> intentionally marked to “To exchange client credentials (client ID >>>>>>>>> and >>>>>>>>> secret) for an access token. This uses the client credentials >>>>>>>>> flow.” >>>>>>>>> [3]. Technically: client ID and secret are submitted using a HTTP >>>>>>>>> POST >>>>>>>>> request to that Iceberg REST endpoint. >>>>>>>>> >>>>>>>>> Having ‘/v1/oauth/tokens’ in the Iceberg REST specification can >>>>>>>>> easily >>>>>>>>> be seen as a hard requirement. In order to implement this in >>>>>>>>> compliance >>>>>>>>> with the OAuth 2.0 spec, that ‘/v1/oauth/tokens’ MUST be the >>>>>>>>> authorization server. If users do not (want to) implement an >>>>>>>>> authorization server, the only way to implement this >>>>>>>>> ‘/v1/oauth/tokens’ >>>>>>>>> endpoint would be to proxy ‘/v1/oauth/tokens’ to the actual >>>>>>>>> authorization server, which means, that this proxy technically >>>>>>>>> becomes a >>>>>>>>> “man in the middle” - knowing both all credentials and all >>>>>>>>> involved tokens. >>>>>>>>> >>>>>>>>> Even if an Iceberg REST server does not implement the >>>>>>>>> ‘/v1/oauth/tokens’ >>>>>>>>> endpoint, it can still receive requests to ‘/v1/oauth/tokens’ >>>>>>>>> containing >>>>>>>>> clear text credentials, if clients are misconfigured (humans do >>>>>>>>> make >>>>>>>>> mistakes) - it’s a non-zero risk - bad actors can >>>>>>>>> implement/intercept >>>>>>>>> that ‘/v1/oauth/tokens’ endpoint and just wait for misconfigured >>>>>>>>> clients to send credentials. >>>>>>>>> >>>>>>>>> Further usages of the REST Catalog API path ‘/v1/oauth/tokens’ are >>>>>>>>> “To >>>>>>>>> exchange a client token and an identity token for a more specific >>>>>>>>> access >>>>>>>>> token. This uses the token exchange flow.” and “To exchange an >>>>>>>>> access >>>>>>>>> token for one with the same claims and a refreshed expiration >>>>>>>>> period >>>>>>>>> This uses the token exchange flow.” Both usages should and can be >>>>>>>>> implemented differently. >>>>>>>>> >>>>>>>>> Apache Iceberg, as a table format project, should recommend >>>>>>>>> protecting >>>>>>>>> sensitive information. But Iceberg should not mandate _how_ that >>>>>>>>> protection is implemented - but the Iceberg REST specification >>>>>>>>> does >>>>>>>>> effectively mandate OAuth 2.0, because other Iceberg REST >>>>>>>>> endpoints do >>>>>>>>> refer/require OAuth 2.0 specifics. Users that want to use other >>>>>>>>> mechanisms, because they are forced to do so by their >>>>>>>>> organization, >>>>>>>>> would be locked out of Iceberg REST. >>>>>>>>> >>>>>>>>> Apache Iceberg should not mandate OAuth 2.0 as the only option - >>>>>>>>> for the >>>>>>>>> sake of openness for the project and flexibility for the server >>>>>>>>> implementations. >>>>>>>>> >>>>>>>>> We think that Apache Iceberg REST Catalog spec should not mandate >>>>>>>>> that a >>>>>>>>> catalog implementation responds to requests to produce Auth Tokens >>>>>>>>> (since the REST spec v1 defines a /v1/tokens endpoint, current >>>>>>>>> implementations have to take deliberate actions when responding to >>>>>>>>> those >>>>>>>>> requests, whether with successful token responses or with “access >>>>>>>>> denied” or “unsupported” responses). >>>>>>>>> >>>>>>>>> We propose the following actions: >>>>>>>>> 1. Immediate mitigation: >>>>>>>>> 1.1. Remove the ‘/v1/oauth/tokens’ endpoint entirely from the >>>>>>>>> Iceberg’s >>>>>>>>> OpenAPI spec w/o replacement. >>>>>>>>> 1.2. As long as OAuth2 is the only mechanism supported by the >>>>>>>>> Iceberg >>>>>>>>> client, make the existing client parameter “oauth2-server-uri” >>>>>>>>> mandatory. The Iceberg REST catalog must fail to initialize if the >>>>>>>>> “oauth2-server-uri” parameter is not defined. >>>>>>>>> 1.3. Remove all fallbacks to the ‘/v1/oauth/tokens’ endpoint. >>>>>>>>> 1.4. Forbid or discourage the communication of tokens from any >>>>>>>>> Iceberg >>>>>>>>> REST Catalog endpoint, both via the "token" property or with any >>>>>>>>> of the >>>>>>>>> "urn:ietf:params:oauth:token-type:*" properties. >>>>>>>>> 2. As a follow up: We’d propose a couple of implementation fixes >>>>>>>>> and >>>>>>>>> changes and test improvements. >>>>>>>>> 3. As a follow up: Define a discovery mechanism for both the >>>>>>>>> Iceberg >>>>>>>>> REST base URI and OAuth 2.0 endpoints/discovery, which allows >>>>>>>>> users to >>>>>>>>> use a single URI to securely access Iceberg REST endpoints. >>>>>>>>> 4. As a follow up: Not new, but we also want to improve the >>>>>>>>> Iceberg REST >>>>>>>>> specification via the “new” REST proposal. >>>>>>>>> >>>>>>>>> We do not think that adding recommendations to >>>>>>>>> inline-documentation is >>>>>>>>> enough to fully mitigate the above concerns. >>>>>>>>> >>>>>>>>> >>>>>>>>> References: >>>>>>>>> >>>>>>>>> [1] RFC 6749 - The OAuth 2.0 Authorization Framework, >>>>>>>>> https://datatracker.ietf.org/doc/html/rfc6749 >>>>>>>>> [2] Iceberg pull request 4771 - Core: Add OAuth2 to REST catalog >>>>>>>>> spec - >>>>>>>>> https://github.com/apache/iceberg/pull/4771 >>>>>>>>> [3] Iceberg pull request 4843 - Spec: Add more context about >>>>>>>>> OAuth2 to >>>>>>>>> the REST spec - https://github.com/apache/iceberg/pull/4843 >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Robert Stupp >>>>>>>>> @snazy >>>>>>>>> >>>>>>>>>
