A server may fail a request, but it should have a specific reason to do so rather than failing randomly. It would be much clearer if the specification explicitly stated this.
Since this discussion started from a question about how to interpret the spec, documenting our consensus directly in the spec would benefit future readers. Otherwise, people will have to search the mailing list or reach out for help to get clarity. WDYT? Yufei On Tue, Oct 13, 2026 at 12:43 AM youngrae kim <[email protected]> wrote: > > The spec PR is up: https://github.com/apache/iceberg/pull/18480 > > It adds "none" to the enum and one sentence defining it; the rest of the > header description is unchanged. > > Youngrae Kim > > 2026년 10월 13일 (화) 오후 4:16, youngrae kim <[email protected]>님이 작성: >> >> Sounds good. >> I will open a spec PR that only adds "none" to the >> X-Iceberg-Access-Delegation enum and leaves the rest of the description as >> is. >> >> Youngrae Kim >> >> 2026년 10월 13일 (화) 오전 7:39, Dmitri Bourlatchkov <[email protected]>님이 작성: >>> >>> Hi All, >>> >>> I agree that we do not need any specific statement on failing requests in >>> this case. >>> >>> Cheers, >>> Dmitri. >>> >>> On Mon, Oct 12, 2026 at 4:12 PM Daniel Weeks <[email protected]> wrote: >>>> >>>> I think we only need to add #1 above. >>>> >>>> For #2, a server may always reject/fail a request. I don't think we need >>>> to clarify that point. >>>> >>>> On Mon, Oct 12, 2026 at 12:46 AM youngrae kim <[email protected]> >>>> wrote: >>>>> >>>>> Thanks everyone. Summarizing where the thread landed: >>>>> >>>>> 1. Add "none" to X-Iceberg-Access-Delegation, meaning the client does not >>>>> need delegated access from the catalog (Dan's suggestion, +1 from Yufei >>>>> and Dmitri). >>>>> 2. Clarify the header description so that failing the request is >>>>> explicitly allowed: "The server may choose to supply access via any or >>>>> none of the requested mechanisms, or fail the request." >>>>> >>>>> Unless someone objects, I am happy to open a spec PR with these two >>>>> changes so the discussion can continue on concrete wording. >>>>> >>>>> Youngrae Kim >>>>> >>>>> 2026년 10월 8일 (목) 오전 8:03, Dmitri Bourlatchkov <[email protected]>님이 작성: >>>>>> >>>>>> Hi Alex, Dan, >>>>>> >>>>>> Thanks for proposing this change. "None" sounds like a good idea to me. >>>>>> It is certainly clearer than "local" in the context of >>>>>> X-Iceberg-Access-Delegation. >>>>>> >>>>>> Cheers, >>>>>> Dmitri. >>>>>> >>>>>> On Mon, Oct 5, 2026 at 4:18 PM Daniel Weeks <[email protected]> wrote: >>>>>>> >>>>>>> Alex, >>>>>>> >>>>>>> Rather than 'local', I think I would prefer 'none'. Rather than saying >>>>>>> what the client is intending to do, express that the client isn't >>>>>>> asking for or doesn't need credentials/access from the catalog. >>>>>>> >>>>>>> At least then, the catalog could skip the generation. >>>>>>> >>>>>>> -Dan >>>>>>> >>>>>>> On Mon, Oct 5, 2026 at 12:51 AM Alex Dutra <[email protected]> wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> How about at least introducing a new enum constant "local" in the >>>>>>>> X-Iceberg-Access-Delegation header? If the intent of that header was >>>>>>>> to express capabilities, this new constant would convey the notion of >>>>>>>> "I can work with my own credentials" more firmly than letting the >>>>>>>> server second-guess. E.g.: "local,remote-signing,vended-credentials" >>>>>>>> would mean the client can either work with vended credentials, remote >>>>>>>> signing, or its own credentials. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Alex >>>>>>>> >>>>>>>> On Tue, Sep 29, 2026 at 4:08 AM youngrae kim <[email protected]> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > Thanks Dan and Alex. >>>>>>>> > >>>>>>>> > Dan, that background is helpful, and I take the point: with >>>>>>>> > storage-credentials and remote-signing-config >>>>>>>> > the catalog can state what it provides, so the spec does not need to >>>>>>>> > mandate a failure when the requested >>>>>>>> > mechanisms cannot be provided. Polaris will keep its current >>>>>>>> > behavior as an implementation choice, >>>>>>>> > documented as such, and I will report that outcome on the Polaris >>>>>>>> > list. >>>>>>>> > >>>>>>>> > The one thing I would still find useful is Alex's narrower point: a >>>>>>>> > sentence saying whether the >>>>>>>> > header is a capability statement or a requirement, and what an >>>>>>>> > absent header means. >>>>>>>> > That does not constrain what the catalog decides to provide; it only >>>>>>>> > tells servers and clients how to read the header. >>>>>>>> > If there is appetite for that I am happy to draft it; otherwise I am >>>>>>>> > fine leaving the wording as is and >>>>>>>> > letting the catalog-driven model evolve with the File type work. >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > Youngrae Kim >>>>>>>> > >>>>>>>> > 2026년 9월 29일 (화) 오전 6:28, Daniel Weeks <[email protected]>님이 작성: >>>>>>>> >> >>>>>>>> >> I'm not sure I agree with the model of a client bringing its own >>>>>>>> >> credentials if the catalog is also responsible for controlling >>>>>>>> >> access (can't really stop people from using that model, but as I >>>>>>>> >> said above, it feels like an anti-pattern). >>>>>>>> >> >>>>>>>> >> The current wording in the spec was intentionally loose so that the >>>>>>>> >> catalog can make decisions without being forced into specific >>>>>>>> >> security/access decisions. I still don't think that updating the >>>>>>>> >> wording would be flexible enough. >>>>>>>> >> >>>>>>>> >> I'm not sure what all catalogs do, but I can say that it's always >>>>>>>> >> safe for the catalog to return credentials/remote config even if >>>>>>>> >> the client was externally configured to use its own. The current >>>>>>>> >> reference implementation bypasses that, I believe, but conceptually >>>>>>>> >> a client does not need to use the mechanism provided by the >>>>>>>> >> catalog. This again feels like a bit of an anti-pattern: if there's >>>>>>>> >> a reason to bring sidecar credentials, it implies a gap in the >>>>>>>> >> delegation model that we should be looking into addressing. >>>>>>>> >> >>>>>>>> >> Dan >>>>>>>> >> >>>>>>>> >> On Mon, Sep 28, 2026 at 10:00 AM Alex Dutra <[email protected]> >>>>>>>> >> wrote: >>>>>>>> >>> >>>>>>>> >>> Hi all, >>>>>>>> >>> >>>>>>>> >>> > The delegation header intended to allow the client to >>>>>>>> >>> > communicate its capabilities, but we've aligned more clearly on >>>>>>>> >>> > catalogs controlling access as opposed to clients defining >>>>>>>> >>> > behavior. >>>>>>>> >>> >>>>>>>> >>> IMO, this header isn't ideal for communicating client capabilities. >>>>>>>> >>> Notably, it fails to represent a key capability: that of using the >>>>>>>> >>> client's own storage credentials to bypass access delegation >>>>>>>> >>> entirely. >>>>>>>> >>> Right now, the only way to imply this capability is by omitting the >>>>>>>> >>> header altogether, which introduces ambiguity into the >>>>>>>> >>> specification. >>>>>>>> >>> Put simply, a client cannot explicitly communicate e.g.: "I can >>>>>>>> >>> function either without access delegation or using vended >>>>>>>> >>> credentials, >>>>>>>> >>> but I do not support remote signing." >>>>>>>> >>> >>>>>>>> >>> Therefore, Apache Polaris today treats this header as an explicit >>>>>>>> >>> requirement rather than just a capability statement. IOW, it treats >>>>>>>> >>> the header presence as an indication that the client cannot >>>>>>>> >>> function >>>>>>>> >>> without access delegation, and that at least one specified mode >>>>>>>> >>> must >>>>>>>> >>> be satisfied. >>>>>>>> >>> >>>>>>>> >>> I would be interested to hear how other catalog implementations are >>>>>>>> >>> approaching this. >>>>>>>> >>> >>>>>>>> >>> In any case, I still think that a minor spec clarification would be >>>>>>>> >>> welcome, if for nothing else, at least to lift the ambiguity around >>>>>>>> >>> what it means when the header is absent and clarify whether the >>>>>>>> >>> header >>>>>>>> >>> is a capability statement, or a requirement. >>>>>>>> >>> >>>>>>>> >>> Thanks, >>>>>>>> >>> Alex >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> On Mon, Sep 28, 2026 at 5:03 PM Daniel Weeks <[email protected]> >>>>>>>> >>> wrote: >>>>>>>> >>> > >>>>>>>> >>> > Youngrae, >>>>>>>> >>> > >>>>>>>> >>> > I think the issue with #2 above is that it's a bit of an >>>>>>>> >>> > anti-pattern for the client to concurrently own access. Either >>>>>>>> >>> > the catalog should be the authority on access or it shouldn't. >>>>>>>> >>> > Security around the physical data shouldn't really be a >>>>>>>> >>> > negotiation between the client and the server. If the catalog >>>>>>>> >>> > is delegating access to the data, it should be the sole source >>>>>>>> >>> > of truth. >>>>>>>> >>> > >>>>>>>> >>> > It feels like we're trying to make this a negotiation between >>>>>>>> >>> > the catalog and client, but the catalog should be responsible >>>>>>>> >>> > for access delegation in a secure environment. >>>>>>>> >>> > >>>>>>>> >>> > The background on this header is important: The original access >>>>>>>> >>> > control mechanism was remote signing only, which was not >>>>>>>> >>> > originally part of the REST protocol (it was separate and S3 >>>>>>>> >>> > specific). We later introduced vended credentials, but only >>>>>>>> >>> > through the config payload. Since then, a lot has changed: we >>>>>>>> >>> > introduced the Storage Credentials and now the Remote Signing >>>>>>>> >>> > Config, which means that we're in a much better position for the >>>>>>>> >>> > catalog to clearly indicate which mechanism the client should >>>>>>>> >>> > use. The mechanisms are more standardized, and clients should >>>>>>>> >>> > be able to report if they cannot support what the catalog has >>>>>>>> >>> > provided. >>>>>>>> >>> > >>>>>>>> >>> > While there may be some residual value in the client signaling >>>>>>>> >>> > its capabilities, I'm not convinced we need to change the >>>>>>>> >>> > wording in the spec and we need to improve the other mechanisms >>>>>>>> >>> > as we evolve to support new functionality like File type, which >>>>>>>> >>> > may mix different modes of access. >>>>>>>> >>> > >>>>>>>> >>> > The behavior you describe in Polaris is perfectly fine as an >>>>>>>> >>> > implementation choice, but I don't think catalogs SHOULD fail if >>>>>>>> >>> > they cannot provide any requested mechanism. The catalog can >>>>>>>> >>> > provide an alternative (e.g. you requested signing, but I'm only >>>>>>>> >>> > providing credentials), which the client can error on if it does >>>>>>>> >>> > not understand. Alternatively, the catalog could reject all >>>>>>>> >>> > requested forms and return only the metadata, effectively >>>>>>>> >>> > signaling that the client can only see the metadata, but has no >>>>>>>> >>> > access to the physical data. >>>>>>>> >>> > >>>>>>>> >>> > -Dan >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > On Fri, Sep 25, 2026 at 8:18 PM youngrae kim >>>>>>>> >>> > <[email protected]> wrote: >>>>>>>> >>> >> >>>>>>>> >>> >> Thanks Alex, Yufei and Dan. >>>>>>>> >>> >> >>>>>>>> >>> >> Dan, I think the catalog-controlled direction and Alex's >>>>>>>> >>> >> wording are compatible, and it may help to separate the two >>>>>>>> >>> >> roles the header plays today: >>>>>>>> >>> >> >>>>>>>> >>> >> 1. Which mechanism to use. >>>>>>>> >>> >> That is already the catalog's decision in Polaris: the server >>>>>>>> >>> >> picks among the mechanisms >>>>>>>> >>> >> it supports for that catalog and storage, and the header only >>>>>>>> >>> >> tells it which ones the client can consume. >>>>>>>> >>> >> Nothing in the proposed sentence changes that. >>>>>>>> >>> >> >>>>>>>> >>> >> 2. Whether the client can access the data at all without >>>>>>>> >>> >> delegation. Today the header is the >>>>>>>> >>> >> only signal a client has for "I have no storage credentials of >>>>>>>> >>> >> my own". >>>>>>>> >>> >> If the catalog cannot provide any mechanism the client can >>>>>>>> >>> >> consume, the request cannot succeed in any >>>>>>>> >>> >> useful sense, and the question is only where the failure >>>>>>>> >>> >> surfaces: at load time with a >>>>>>>> >>> >> message naming the cause, or at the first data access with a >>>>>>>> >>> >> storage error. >>>>>>>> >>> >> >>>>>>>> >>> >> The proposed SHOULD covers exactly the empty-intersection case >>>>>>>> >>> >> in (2); a server that knows its >>>>>>>> >>> >> clients carry their own credentials can still return the table, >>>>>>>> >>> >> which is why SHOULD rather than MUST seems right. >>>>>>>> >>> >> >>>>>>>> >>> >> If the longer-term direction is to rely less on the header, I >>>>>>>> >>> >> would be interested in how a >>>>>>>> >>> >> catalog-driven model would express (2), for example the client >>>>>>>> >>> >> learning from /v1/config which >>>>>>>> >>> >> mechanisms the catalog offers, so it can decide before loading >>>>>>>> >>> >> a table. That seems like a >>>>>>>> >>> >> larger change than the clarification this thread started with, >>>>>>>> >>> >> so I would suggest landing the >>>>>>>> >>> >> one-sentence clarification for the current header and taking >>>>>>>> >>> >> the catalog-driven model as a >>>>>>>> >>> >> separate discussion, unless you see a conflict between the two. >>>>>>>> >>> >> >>>>>>>> >>> >> Thanks, >>>>>>>> >>> >> Youngrae Kim >>>>>>>> >>> >> >>>>>>>> >>> >> 2026년 9월 26일 (토) 오전 8:52, Daniel Weeks <[email protected]>님이 작성: >>>>>>>> >>> >>> >>>>>>>> >>> >>> I'm not convinced that we're thinking about this the right way. >>>>>>>> >>> >>> >>>>>>>> >>> >>> The delegation header intended to allow the client to >>>>>>>> >>> >>> communicate its capabilities, but we've aligned more clearly >>>>>>>> >>> >>> on catalogs controlling access as opposed to clients defining >>>>>>>> >>> >>> behavior. >>>>>>>> >>> >>> >>>>>>>> >>> >>> If we leaned more into that direction, then access would be >>>>>>>> >>> >>> determined by the catalog based on the availability of vented >>>>>>>> >>> >>> credentials or the new remote signing configuration. >>>>>>>> >>> >>> >>>>>>>> >>> >>> There may be an opportunity here to rely less on the header >>>>>>>> >>> >>> and give responsibility back on th e catalog to determine the >>>>>>>> >>> >>> access model. >>>>>>>> >>> >>> >>>>>>>> >>> >>> -Dan >>>>>>>> >>> >>> >>>>>>>> >>> >>> On Fri, Sep 25, 2026, 4:39 PM Yufei Gu <[email protected]> >>>>>>>> >>> >>> wrote: >>>>>>>> >>> >>>> >>>>>>>> >>> >>>> I tend to agree with Alex and Youngrae. A fail-fast approach >>>>>>>> >>> >>>> provides a clear behavioral contract between the server and >>>>>>>> >>> >>>> clients, while the current spec introduces ambiguity. >>>>>>>> >>> >>>> >>>>>>>> >>> >>>> I am not sure if this ambiguity was intentional. Please chime >>>>>>>> >>> >>>> in if you can provide more context; otherwise, I suggest we >>>>>>>> >>> >>>> update the spec to eliminate it. >>>>>>>> >>> >>>> >>>>>>>> >>> >>>> Yufei >>>>>>>> >>> >>>> >>>>>>>> >>> >>>> >>>>>>>> >>> >>>> On Thu, Sep 24, 2026 at 10:12 AM Alex Dutra >>>>>>>> >>> >>>> <[email protected]> wrote: >>>>>>>> >>> >>>>> >>>>>>>> >>> >>>>> Hi Youngrae, >>>>>>>> >>> >>>>> >>>>>>>> >>> >>>>> Thanks for starting this thread! >>>>>>>> >>> >>>>> >>>>>>>> >>> >>>>> I believe the phrase "via any or none of the requested >>>>>>>> >>> >>>>> mechanisms" >>>>>>>> >>> >>>>> requires clarification. >>>>>>>> >>> >>>>> >>>>>>>> >>> >>>>> If a client requests modes A and B, a server returning only >>>>>>>> >>> >>>>> mode B is >>>>>>>> >>> >>>>> perfectly acceptable. However, returning a 200 OK response >>>>>>>> >>> >>>>> with >>>>>>>> >>> >>>>> neither mode A nor B represents an ambiguous outcome. >>>>>>>> >>> >>>>> >>>>>>>> >>> >>>>> The current wording likely assumes the client could >>>>>>>> >>> >>>>> theoretically >>>>>>>> >>> >>>>> proceed using its own fallback storage credentials. In >>>>>>>> >>> >>>>> practice, >>>>>>>> >>> >>>>> operators configure clients intentionally; requesting access >>>>>>>> >>> >>>>> delegation "just in case" is unrealistic. A fail-fast error >>>>>>>> >>> >>>>> is a >>>>>>>> >>> >>>>> better outcome, as opposed to letting the client proceed, >>>>>>>> >>> >>>>> only to see >>>>>>>> >>> >>>>> it fail later on. >>>>>>>> >>> >>>>> >>>>>>>> >>> >>>>> I propose updating the specification to mandate an error >>>>>>>> >>> >>>>> response when >>>>>>>> >>> >>>>> no requested mechanism is available, while also clarifying >>>>>>>> >>> >>>>> whether >>>>>>>> >>> >>>>> providing multiple delegation modes is supported: >>>>>>>> >>> >>>>> >>>>>>>> >>> >>>>> "The server MAY supply access via any or all of the requested >>>>>>>> >>> >>>>> mechanisms. If none of the requested mechanisms is >>>>>>>> >>> >>>>> available, the >>>>>>>> >>> >>>>> server SHOULD return an HTTP 400 (Bad Request) response." >>>>>>>> >>> >>>>> >>>>>>>> >>> >>>>> Thanks, >>>>>>>> >>> >>>>> Alex >>>>>>>> >>> >>>>> >>>>>>>> >>> >>>>> On Wed, Sep 23, 2026 at 11:14 AM youngrae kim >>>>>>>> >>> >>>>> <[email protected]> wrote: >>>>>>>> >>> >>>>> > >>>>>>>> >>> >>>>> > Hi all, >>>>>>>> >>> >>>>> > The REST catalog spec describes >>>>>>>> >>> >>>>> > X-Iceberg-Access-Delegation as an "optional signal to the >>>>>>>> >>> >>>>> > server that the client supports delegated access via a >>>>>>>> >>> >>>>> > comma-separated list of access mechanisms. The server may >>>>>>>> >>> >>>>> > choose to supply access via any or none of the requested >>>>>>>> >>> >>>>> > mechanisms." [1] >>>>>>>> >>> >>>>> > >>>>>>>> >>> >>>>> > In Apache Polaris we recently discussed what a server >>>>>>>> >>> >>>>> > should do when the header is present but it can satisfy >>>>>>>> >>> >>>>> > none of the requested mechanisms, for example >>>>>>>> >>> >>>>> > vended-credentials against an S3-compatible store without >>>>>>>> >>> >>>>> > STS, or remote-signing on a server that does not implement >>>>>>>> >>> >>>>> > it [2]. >>>>>>>> >>> >>>>> > >>>>>>>> >>> >>>>> > Polaris fails such requests with HTTP 400 and a message >>>>>>>> >>> >>>>> > naming the reason, because in these cases the inability is >>>>>>>> >>> >>>>> > a static property of the catalog and a 200 without >>>>>>>> >>> >>>>> > credentials would only move the failure to the first data >>>>>>>> >>> >>>>> > access. >>>>>>>> >>> >>>>> > >>>>>>>> >>> >>>>> > Two readings of "any or none" came up and the text does >>>>>>>> >>> >>>>> > not settle which is intended: >>>>>>>> >>> >>>>> > (a) "none" means the server returns 200 with the table >>>>>>>> >>> >>>>> > and no delegated access, and the client detects this from >>>>>>>> >>> >>>>> > the response: an empty storage-credentials (and, we >>>>>>>> >>> >>>>> > understand, an empty remote-signing-config once that field >>>>>>>> >>> >>>>> > lands in 1.12), falling back to the config map for older >>>>>>>> >>> >>>>> > servers [3]. >>>>>>>> >>> >>>>> > >>>>>>>> >>> >>>>> > (b) "none" also permits the server to reject the request >>>>>>>> >>> >>>>> > when it cannot provide any of the requested mechanisms. >>>>>>>> >>> >>>>> > >>>>>>>> >>> >>>>> > Questions for the community: >>>>>>>> >>> >>>>> > 1. Is rejecting the request a permitted server choice >>>>>>>> >>> >>>>> > under the current text, or is (a) the only intended >>>>>>>> >>> >>>>> > behavior? >>>>>>>> >>> >>>>> > 2. If rejection is permitted, is there a preferred >>>>>>>> >>> >>>>> > status code and error type, so clients can distinguish >>>>>>>> >>> >>>>> > "delegation unavailable" from other 400s? >>>>>>>> >>> >>>>> > 3. If (a) is the intended behavior, should the spec >>>>>>>> >>> >>>>> > state explicitly that the absence of credentials in both >>>>>>>> >>> >>>>> > storage-credentials and config (and of >>>>>>>> >>> >>>>> > remote-signing-config) is the signal that no delegation >>>>>>>> >>> >>>>> > was applied? >>>>>>>> >>> >>>>> > >>>>>>>> >>> >>>>> > I'm happy to draft a clarification to the parameter >>>>>>>> >>> >>>>> > description once there is agreement. >>>>>>>> >>> >>>>> > >>>>>>>> >>> >>>>> > [1] >>>>>>>> >>> >>>>> > https://github.com/apache/iceberg/blob/778d103c0fa802be5e85e4cb2e4e4b4bcd8ac891/open-api/rest-catalog-open-api.yaml#L2151-L2161 >>>>>>>> >>> >>>>> > [2] >>>>>>>> >>> >>>>> > https://lists.apache.org/thread/9oyv4v0vk3t8stv9832npl8t0yyp4j8r >>>>>>>> >>> >>>>> > [3] >>>>>>>> >>> >>>>> > https://github.com/apache/iceberg/blob/778d103c0fa802be5e85e4cb2e4e4b4bcd8ac891/open-api/rest-catalog-open-api.yaml#L4246-L4249 >>>>>>>> >>> >>>>> > >>>>>>>> >>> >>>>> > Thanks, >>>>>>>> >>> >>>>> > Youngrae Kim
