Hey William, I had a chance to sync with Russell on this topic and I agree that Azure has some unique differences that make the current remote signing difficult.
In terms of how to best support the pre-signing models, I believe that we can actually accommodate that implementation under the current remote signing APIs. Back when we designed the remote signing APIs, we considered the case where the signer wanted to return a different path than what was originally requested (primarily for cases like routing/redirection). However, that also allows for the catalog to return a presigned URI instead of the originally requested path. The endpoint itself is general (just `/sign`) and adding a new option to access delegation would be a good signal for what a client is requesting. In terms of use cases, there are two different scenarios: 1. Where a scan/plan request returns presigned URLs: in this case the FileIO should detect the presigned url and execute it directly instead of using the SDK client. 2. The normal planning mode, where the FileIO should contact the signing service instead of going directly through the SDK to get a presigned URL. I believe both of these scenarios can be achieved by introducing a common BaseInputFile/BaseInputStream that shares the identification or redirection logic for presigning. Does this make sense? I'll also follow up on the PR, but I think there's very little we need to do in order to accommodate both paths. -Dan On Fri, Aug 14, 2026 at 1:43 PM William Hyun <[email protected]> wrote: > Here is a POC for following along with this claim about Azure not > supporting S3-style remote signing natively. > - https://github.com/williamhyun/Azure-SAS-Header-POC > > On 2026/07/21 20:01:16 William Hyun wrote: > > Hi All, > > > > Following up on the ongoing discussion thread regarding the file-level > access delegation modes in the Iceberg REST spec, I wanted to share some > findings from a cross-cloud perspective of how the major Cloud Service > Providers handle these protocols. > > > > When evaluating remote signing as an access mode, two major operational > hurdles emerge: strict 15-minute Time-To-Live constraints and a lack of > native support across all CSPs, particularly Azure. > > > > 1. Non-Configurable TTLs > > AWS: In the AWS SigV4 protocol, the calculated signature with a > timestamp inherently expires 15 minutes after. [1] > > GCS: GCS enforces an identical constraint for remote header signing. A > signature passed via request headers is valid only from 15 minutes prior to > the active datetime until exactly 15 minutes after. [2] > > > > 2. Lack of Native Remote Signing Support in Azure > > While AWS (SigV4) and GCS (V4) natively accept short-lived, granular > signatures via the Authorization header, Microsoft Azure Blob Storage and > ADLS Gen2 fundamentally reject this paradigm. > > > > In Azure Storage, a Shared Access Signature (SAS) is strictly a token > designed to be appended to the resource URI as query parameters. [3] > > To execute true remote header signing for Azure Storage, a catalog would > be forced to use the legacy SharedKey authorization scheme, formatting the > header as Authorization: SharedKey <AccountName>:<Signature>. Constructing > this signature mathematically requires the catalog to hold the storage > account's master access key. [4] [5] > > Because of this, Microsoft explicitly recommends disallowing SharedKey > authorization entirely, advising modern architectures to rely instead on > User Delegation SAS tokens (via the URI) or Microsoft Entra ID. [5] > > > > On the other hand, pre-signed URLs are natively and uniformly supported > across all three major CSPs. While the underlying cryptographic signatures > and generation methods differ (AWS SigV4, GCS V4, Azure SAS), the resulting > access artifact is universally a standard, fetchable HTTPS URL. > Furthermore, pre-signed URLs allow for highly customizable expiration > times. The maximum TTL is upper bound by the validity of the signing > credential itself (e.g. temporary STS credentials which are default 1-hour > sessions but can be configured up to 12 hours). This provides significantly > more operational flexibility than the rigid 15-minute window enforced on > header-based signatures. > > > > Given the findings above, I would still like to discuss the possibility > of pre-signed URLs. > > During our last sync, a valid concern was raised: our current SDKs and > FileIO implementations don't support processing raw, pre-signed HTTPS URLs. > Thus requiring a new, separate read path to be built. To help us evaluate > exactly what this effort entails and to discuss the trade-offs, I’ve put > together a POC for what this pre-signed URL read path might look like: > > https://github.com/williamhyun/iceberg/pull/206 > > > > Looking forward to hearing your thoughts! > > > > Best, > > William > > > > [1] > https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-api-request-authentication.html#sqs-api-basic-authentication-process > > [2] > https://docs.cloud.google.com/storage/docs/authentication/signatures#active_datetime > > [3] > https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview#how-a-shared-access-signature-works > > [4] > https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#specifying-the-authorization-header > > [5] > https://learn.microsoft.com/en-us/azure/storage/common/authorize-data-access?tabs=blobs#authorization-for-data-operations > > > > > > On 2026/07/08 01:16:35 William Hyun wrote: > > > Hi all, > > > > > > Following up on the earlier discussion around file-level access > delegation > > > in the REST Catalog spec, I put together a small POC PR against my > personal > > > Iceberg fork to explore the remote signing + caching direction we > discussed: > > > > > > https://github.com/williamhyun/iceberg/pull/198 > > > > > > The goal is to make the tradeoffs more concrete and evaluate whether > this > > > could be a practical approach for finer-grained delegated access. > > > Feedback is very welcome. > > > > > > Best, > > > William > > > > > > > > > > > > On Thu, Jul 2, 2026 at 5:29 PM William Hyun <[email protected]> > wrote: > > > > > > > Hi all, > > > > > > > > Thank you again for the discussion during the 7/1 catalog community > sync. > > > > I wanted to summarize two alternative directions that came up to > > > > pre-signed URLs: > > > > 1. scoped credential vending > > > > 2. remote signing with caching and bulk-signing > > > > > > > > I think scoped credential vending is worth considering in cases > where the > > > > authorization boundary maps cleanly to a small number of stable > storage > > > > prefixes. > > > > Scoped credential vending seems less practical when access needs to > be > > > > granted for many specific files rather than a small number of > prefixes. > > > > > > > > In AWS, session policies are passed as parameters during > AssumeRole-style > > > > issuance when the temporary session is created. [1] > > > > This means that scoping down an already-issued credential cannot be > done > > > > locally by simply altering the existing token. > > > > Instead, it requires minting another STS session whose policy > encodes the > > > > new scope. > > > > This turns authorization into an STS issuance problem with > limitations on > > > > STS request frequency [2] and policy size. [3] > > > > > > > > By contrast, pre-signed URLs and remote signing can authorize exact > object > > > > requests using already-held credentials, without creating a new STS > session > > > > for each file set. > > > > Based on that discussion, I am exploring a POC around bulk remote > > > > signing + caching to evaluate whether it could be a practical > alternative > > > > to pre-signed URLs for these finer-grained access patterns. > > > > > > > > Best, > > > > William > > > > > > > > [1] > https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html > > > > [2] > > > > > https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html#reference_iam-quotas-sts-requests > > > > [3] > > > > > https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html > > > > > > > > On Tue, Jun 30, 2026 at 7:43 PM William Hyun <[email protected]> > wrote: > > > > > > > >> Hi everyone, > > > >> > > > >> I'd like to open a discussion on an extension to the REST Catalog > spec to > > > >> enable file-level access delegation during scan planning. > > > >> Today, delegated access is table-scoped, which can force > > > >> over-provisioning or table fragmentation when consumers should only > see a > > > >> subset of partitions or records. > > > >> This proposal adds a path using pre-signed URLs to make > partition-scoped > > > >> sharing practical in the REST Catalog model. > > > >> > > > >> Here is my proposal: https://s.apache.org/n16st > > > >> > > > >> I look forward to hearing your thoughts. > > > >> > > > >> Relevant discussion: > > > >> - https://lists.apache.org/thread/ko9kp0gvzhx85n7cvoxqnpw4vwnhmdg6 > > > >> > > > >> Best, > > > >> William > > > >> > > > >> > > > > > >
