Hi Alex, > What the recorded decision prevents is a principal with > read-only access deleting files once the table is gone. So some > decision about the principal is needed.
Do not get me wrong. I fully support recording the user's allows operations in the token. What I meant in my earlier email is that the time frame when the token is valid beyond the entity's lifetime can be decided without the Authorizer in the loop. For example, a sample config setting might work. WDYT? So a read-only user can never use the token to sign delete requests. However, a user with basic drop privileges could still get a longer term token without being granted a server-side "PURGE" permission, or a dedicated permission for DELETE_FILES_AFTER_DROP. That said, it may be a moon point. My impression is that the discussion on the soft-delete thread [1] to moving towards performing authZ checks on the entity for every signing request. Cheers, Dmitri. [1] https://lists.apache.org/thread/xf4kmddfkv5o5f1qmzxtmz23z4vtdn0x On Tue, Nov 17, 2026 at 5:30 AM Alex Dutra via dev <[email protected]> wrote: > Hi all, > > @Tommy: > > Thank you for confirming that the decision cache makes sense for your > use case, and for your input on the drop-with-purge vs soft-delete > discussion. > > I am finalizing some benchmarks on AWS, I will share the numbers > shortly. After that, I think we can proceed to opening the first PR! > > @Dmitri: > > > However, can those checks slow down the signing process? > > Not really. The overlap check only runs on the staged-create and > dropped-table paths. Sign requests for existing tables don't run such > checks. > > > extending the validity of the signing token beyond the lifetime of the > catalog entity does not look like an authorization decision to me. [...] > However, using the authorizer for this with a different operation name > (e.g. DELETE_FILES_AFTER_DROP) works too. > > I agree that the deletes themselves can be done by any principal with > write access. What the recorded decision prevents is a principal with > read-only access deleting files once the table is gone. So some > decision about the principal is needed. > > I also agree that DROP_TABLE_WITH_PURGE isn't the right operation for > it: the server performs no purge here. A dedicated operation such as > your DELETE_FILES_AFTER_DROP is clearer. > > > This feels more like a catalog-level flag to me with the value being the > maximum time period allowed for use on deleted entities. > > The token's lifespan is already bounded (24 hours by default). This > setting is global currently, and applied for both staged-creates and > purged tables; I'd rather not add a separate, per-realm or per-catalog > setting for now. Do you feel strongly about the catalog-level > override? > > > I think leveraging the soft-delete feature to support Spark use cases is > more robust and easier for users to manage. > > Agreed: a per-request overlap check can't exclude a table being > created at the same location concurrently, while a soft-deleted entity > keeps its hold on the locations. When soft delete is enabled, Polaris > should authorize these requests against the retained entity. Since > soft delete is opt-in, though, I'd keep the token-based path as the > fallback for catalogs that don't enable it. > > @Yufei: > > > engine side purging seems related to staging creation: signing needs to > work when the table either no longer exists or has not been committed yet. > > Exactly, and the token handles both the same way: it carries what the > missing entity can't provide. > > > Could you elaborate on how signing tokens work with staged table > creation out of the box? > > Here is the sequence with an Iceberg 1.12 client: > > 1. The client creates the table with stage-create. Polaris authorizes > the staged create on the namespace, validates the table's locations, > and returns the usual load-table response. That response includes a > remote signing token. The token records the table identifier, its > locations, that the operation is a staged create, the principal, and > an expiry. > > 2. The client calls the sign endpoint and echoes the token in the > request. The endpoint first tries to authorize against the table. > Since there's no table, it falls back to the token: the token must be > valid and mark a staged create. The request is then authorized against > the parent namespace, on every request, with the caller's current > privileges: the token only supplies the locations, not the decision. > > 3. Once the client commits the staged table, subsequent requests are > authorized against the table itself, rather than from the token. > > Note: clients older than Iceberg 1.12 don't send the token, so staged > creates with remote signing are refused for them. That will be > documented as a limitation. > > > One possible workaround for purge would be for the engine to enumerate > the files and obtain signatures for the delete requests before dropping the > table from the catalog > > I see 2 obstacles: the number of files to keep in memory can be huge, > and S3 refuses a request if the x-amz-date header is more than 15 > minutes old. The engine would have to enumerate the files, read the > manifests, sign the deletes, then execute them within 15 minutes. > Also, this needs the Iceberg client changes you mention, whereas the > token approach works with Spark as it is today. > > So, concretely, I'd change my current prototype as follows: > > - rename the recorded decision to a dedicated operation > (DELETE_FILES_AFTER_DROP); > - document that the token lifespan bounds both the staged-create and > post-drop windows; > - document that soft deletes, if enabled, should be preferred over > token-based authz. > > Does that sound reasonable to everybody? AWS numbers coming soon. > > Thanks, > Alex > > > On Tue, Nov 17, 2026 at 3:34 AM Yufei Gu <[email protected]> wrote: > > > > Tommy, thanks for the context. That's helpful. > > > > Alex, engine side purging seems related to staging creation: signing > needs to work when the table either no longer exists or has not been > committed yet. One possible workaround for purge would be for the engine to > enumerate the files and obtain signatures for the delete requests before > dropping the table from the catalog, then execute those deletes after the > drop succeeds. In that model, I think a bounded cleanup window is > reasonable. Credential vending already has a similar limitation: existing > credentials can outlive the table, but refreshing them through the table > endpoint stops working after the drop. The TTL may not be enough for a > large table, but this limitation applies to both. We need to modify the IRC > clients to make this work, though. > > > > Could you elaborate on how signing tokens work with staged table > creation out of the box? I understand your updated proposal carries the > staged locations in the token and checks permissions on the parent > namespace. What I’m missing is how the signing endpoint is accessed when a > table does not exist yet. During staged creation, you can not pre-create > all files, engines must hit the sign endpoint constantly. > > > > Yufei > > > > > > On Mon, Nov 16, 2026 at 11:49 AM Tommy Bollhofer < > [email protected]> wrote: > >> > >> Hi Alex, Yufei and Dmitri, > >> > >> Thanks for the detailed answers, Alex. Replies to the open points: > >> > >> - DuckDB: from what I can see it does not support remote signing today. > The Iceberg extension's ACCESS_DELEGATION_MODE option only accepts > vended_credentials (the default) and none [1]. Support is tracked in > duckdb-iceberg #670 (open since January; a maintainer said it is wanted but > lower priority) [2], and a community pull request, #1318, adds a > remote_signing mode through an Iceberg-owned filesystem; it is still open, > and the maintainers asked whether it can be done in httpfs instead [3]. > >> > >> - Persistence (Yufei): we use the relational JDBC persistence on > PostgreSQL. If it helps your Kubernetes benchmark, that is the setup I can > test against. > >> > >> - PyIceberg: we run PyIceberg 0.11.1 against our catalog, with the s3fs > extra installed, so the fsspec FileIO is available. Once a build exists I > will test it with py-io-impl=pyiceberg.io.fsspec.FsspecFileIO on existing > tables and report back what works with an S3-compatible store. > >> > >> - Spark: our compaction runs on Spark with the Iceberg 1.10 runtime, so > it would be in compat mode (existing tables only). Good to know that works; > I can try the 1.12 runtime for a test if that is useful. > >> > >> - Decision cache: from a user's side, a configurable TTL that is off by > default, with the maximum revocation delay documented as part of the > contract, is fine for us. > >> > >> - Drop with purge: we run with table purge disabled and delete files > outside the catalog, so we have no stake in the post-drop signing question > and no preference between the token-recorded decision and the soft-delete > approach. > >> > >> Best, > >> Tommy > >> > >> [1] > https://duckdb.org/docs/stable/core_extensions/iceberg/iceberg_options > (ACCESS_DELEGATION_MODE: "Allowed values are vended_credentials and none") > >> [2] https://github.com/duckdb/duckdb-iceberg/issues/670 > >> [3] https://github.com/duckdb/duckdb-iceberg/pull/1318 > >> > >> > >> ________________________________________ > >> From: Dmitri Bourlatchkov <[email protected]> > >> Sent: Friday, November 13, 2026 18:19 > >> To: [email protected] <[email protected]> > >> Cc: Yufei Gu <[email protected]>; Tommy Bollhofer < > [email protected]>; Alex Dutra <[email protected]> > >> Subject: Re: [DISCUSS] Resume remote signing work > >> > >> > >> Hi Alex, > >> > >> Re: the DROP_TABLE_WITH_PURGE proposal: > >> > >> It looks like it should work with the location overlap checks that you > mentioned. However, can those checks slow down the signing process? > >> > >> Assuming the performance is acceptable, I think it is an interesting > approach and is worth exploring. > >> > >> DROP_TABLE_WITH_PURGE protects server-side actions invoked via > dedicated API methods / parameters. > >> > >> This use case is different in that all deletes are performed on the > client side. Any client with normal write access can perform these deletes > before dropping the table from the catalog. So extending the validity of > the signing token beyond the lifetime of the catalog entity does not look > like an authorization decision to me. Since we validate that locations do > not overlap, signed requests will stay within the scope of the original > entity anyway. > >> > >> This feels more like a catalog-level flag to me with the value being > the maximum time period allowed for use on deleted entities. However, using > the authorizer for this with different operation name (e.g. > DELETE_FILES_AFTER_DROP) works too. > >> > >> Re: soft delete: > >> > >> I think leveraging the soft-delete feature to support Spark use cases > is more robust and easier for users to manage. > >> > >> The soft-deleted entity still maintains a hold on table locations. I > think this offers a stronger protection against overlaps than one-off > checks in each signing request (check/create races are still possible). > >> > >> Undropping the table could check if (some key) files are still present. > >> > >> And yes, we can indeed offer the option of denying engines all access > to soft-deleted table files. > >> > >> Cheers, > >> Dmitri. > >> > >> On Fri, Nov 13, 2026 at 5:15 PM Alex Dutra via dev < > [email protected]> wrote: > >> Hi all, > >> > >> One more remote signing question, raised while testing with Spark. > >> > >> Spark implements DROP TABLE ... PURGE client-side: it loads the table, > >> drops it without purging it, then deletes its files itself, with the > >> FileIO of the loaded table [1]. With vended credentials this works, as > >> the credentials outlive the drop. With per-request authorization, each > >> sign request is authorized against a table that no longer exists, so > >> the files are left behind. > >> > >> Proposal: when a table is loaded, created or registered, Polaris also > >> asks the authorizer whether the caller is allowed to > >> DROP_TABLE_WITH_PURGE, and records the outcome in the remote > >> signing token, along with the table's ID and locations. Once the table > is > >> dropped, sign requests carrying a valid token are allowed if the > recorded > >> decision allows it, the request only reads or deletes objects at the > >> recorded locations, > >> the table was dropped rather than renamed, and no other table or > >> namespace now uses those locations. Requests without a token are > >> rejected. > >> > >> Requests for tables that exist are still authorized per request, with > >> or without a token. The decision is only used once the table's grants > >> are gone with it, and the authorizer (including OPA and Ranger) makes > >> it, at load time. Spark loads the table right before purging it, so > >> the decision is recent. > >> > >> The soft-delete proposal [2] would let Polaris authorize such requests > >> against the retained table instead, when enabled. Note that Spark's > >> purge is a drop without purge followed by client-side deletes, so with > >> soft delete on, Spark would delete the files of a table that can still > >> be undropped, unless the catalog refuses it. Remote signing would make > >> refusing possible. > >> > >> Thoughts? > >> > >> Thanks, > >> Alex > >> > >> [1] > https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java > >> [2] https://lists.apache.org/thread/1djyz0x3k7m71d71bqhw11c3syyshtdr > >> > >> > >> On Wed, Nov 11, 2026 at 9:21 AM Alex Dutra <[email protected]> wrote: > >> > > >> > Hi Yufei, > >> > > >> > Yes I think the cache is reasonable as long as it becomes part of the > >> > security constraints and is clearly documented. > >> > > >> > However, I would not take that -19% as granted: my benchmarks were on > >> > my local laptop only, where network calls to Postgres are cheap. I > >> > still need to reproduce the benchmark on k8s on some cloud provider, > >> > which I will shortly. But I expect those -19% to become worse, in > >> > which case, the cache may turn out to be an essential part of the > >> > setup. > >> > > >> > Thanks, > >> > Alex > >> > > >> > On Tue, Nov 10, 2026 at 7:24 PM Yufei Gu <[email protected]> > wrote: > >> > > > >> > > On second thought, I believe invalidating the decision cache is > hard even with native RBAC, potentially making the system really > complicated. Given that OPA/Ranger cannot even support it, we'd prefer the > simpler solution(TTL). > >> > > > >> > > It looks like other systems provide the same decision cache with > certain constraints. However, A cache is acceptable only when its maximum > revocation delay is an explicit part of the security contract. AWS Verified > Permissions[1] caches authorization decisions for 120 seconds by default, > and says policy changes may not take effect until the cache expires. > OpenFGA has caching off by default and offers a higher consistency mode > that bypasses it. > >> > > > >> > > I think we should make the decision cache configurable and off by > default. When it is on, we must explicitly state the maximum revocation > delay as part of the security contract. > >> > > > >> > > Tommy, would you mind sharing which persistence you are using? I > think -19% for JDBC isn't too bad as the initial version, given that both > communities(Iceberg, Polaris) are actively working on batch signing. > >> > > > >> > > 1. > https://docs.aws.amazon.com/verifiedpermissions/latest/userguide/policy-stores-api-userpool-considerations-troubleshooting.html > >> > > > >> > > Yufei > >> > > > >> > > > >> > > > >> > > On Mon, Nov 9, 2026 at 11:36 AM Alex Dutra via dev < > [email protected]> wrote: > >> > >> > >> > >> Hi Tommy, > >> > >> > >> > >> Thanks for offering to test! > >> > >> > >> > >> Before answering your questions, I forgot to say that I am also > >> > >> planning to introduce a "compatibility mode" for older clients: > this > >> > >> mode is automatically activated when there is no signing token in > the > >> > >> request. Then Polaris would proceed as follows: > >> > >> > >> > >> - Sign requests without a token, on an existing table: accepted. > >> > >> - Sign requests without a token, on a missing table: rejected. > >> > >> > >> > >> In this mode, older clients can read and write existing tables, but > >> > >> get a clear error on CTAS (staged-creates). I think that strikes a > >> > >> good balance between support for older clients, while also taking > >> > >> security seriously by rejecting staged-creates. > >> > >> > >> > >> With that in mind: > >> > >> > >> > >> - Spark: with the Iceberg 1.12 runtime, all operations will be > >> > >> supported. With the 1.10 runtime, we would be in compat mode, and > >> > >> reads and writes on existing tables should work too, but not CTAS. > >> > >> > >> > >> - PyIceberg: should work for existing tables too, with the same > >> > >> limitation, but only with its fsspec FileIO > >> > >> (py-io-impl=pyiceberg.io.fsspec.FsspecFileIO); the default PyArrow > >> > >> FileIO does not support remote signing. I haven't tested it yet, so > >> > >> your feedback would be welcome. > >> > >> > >> > >> - Trino: no. Trino accesses S3 through its own file system, which > does > >> > >> not implement remote signing, whatever the Iceberg library version. > >> > >> > >> > >> - DuckDB: no; as far as I know, its Iceberg extension does not > support > >> > >> remote signing. Could you confirm please? > >> > >> > >> > >> I'll let you know when a build is available. > >> > >> > >> > >> Thanks, > >> > >> Alex > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> On Mon, Nov 9, 2026 at 4:13 PM Tommy Bollhofer < > [email protected]> wrote: > >> > >> > > >> > >> > Hi Alex, Yufei and Dmitri, > >> > >> > > >> > >> > Thanks for the write-up and the numbers, Alex. Some input from a > user of a FlashBlade (S3-compatible, no AssumeRole) deployment, where every > client today holds static storage keys: > >> > >> > > >> > >> > - Per-request authorization with the short decision cache as the > default, and signing tokens only where staged creates need them, sounds > right to us. Our workloads are short, and a revocation taking effect within > the cache TTL (30 seconds) is fine for us. We would not want a no-cache > mode to be the default, given the numbers. > >> > >> > - On TABLE_REMOTE_SIGN: either way works for us, because our > data-writer roles already hold CATALOG_MANAGE_CONTENT (which would imply > it). We have no objection to dropping it and relying on TABLE_READ_DATA / > TABLE_WRITE_DATA plus the catalog-level REMOTE_SIGNING_ENABLED flag, which > keeps the privilege model simpler. > >> > >> > - A 24-hour default for the signing token, configurable, is fine > for us. > >> > >> > > >> > >> > One question: Do you have a view on which engines and clients > will be able to use remote signing in the first release (Trino, Spark, > PyIceberg, DuckDB)? Our stack is Trino with Iceberg 1.12, Spark on the > Iceberg 1.10 runtime (we can move it up for a test), PyIceberg and DuckDB. > We would test whichever of them support it as soon as a build is available, > and report back what works on FlashBlade. > >> > >> > > >> > >> > Best, > >> > >> > Tommy > >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > ________________________________________ > >> > >> > From: Alex Dutra via dev <[email protected]> > >> > >> > Sent: Monday, November 09, 2026 09:16 > >> > >> > To: [email protected] <[email protected]> > >> > >> > Cc: Dmitri Bourlatchkov <[email protected]>; Alex Dutra < > [email protected]> > >> > >> > Subject: Re: [DISCUSS] Resume remote signing work > >> > >> > > >> > >> > > >> > >> > Hi Yufei, Dmitri, > >> > >> > > >> > >> > > >> > >> > > >> > >> > Thanks for the feedback! > >> > >> > > >> > >> > > >> > >> > > >> > >> > @Yufei: > >> > >> > > >> > >> > > >> > >> > > >> > >> > The NoSQL overhead is due to cache deserialization. The NoSQL > cache > >> > >> > > >> > >> > was designed to be GC friendly and holds serialized entities; > but in a > >> > >> > > >> > >> > path that is mostly CPU bound like the sign endpoint, > deserialization > >> > >> > > >> > >> > quickly becomes the bottleneck, even with zero database calls. > We are > >> > >> > > >> > >> > aware of that, and already have a few ideas to improve its > efficiency. > >> > >> > > >> > >> > > >> > >> > > >> > >> > Cached decisions: we could invalidate the cache on permission > changes, > >> > >> > > >> > >> > but that would only work for built-in RBAC indeed. For OPA or > Ranger, > >> > >> > > >> > >> > revocations would take effect within the cache TTL. That's why I > think > >> > >> > > >> > >> > the cache TTL should be low: 30 seconds by default. I can > benchmark > >> > >> > > >> > >> > shorter TTLs. That's also why the cache should only be applied > to the > >> > >> > > >> > >> > sign endpoint, not to others. > >> > >> > > >> > >> > > >> > >> > > >> > >> > “Signing tokens”: the benchmarks showed that per-request authz > remains > >> > >> > > >> > >> > doable performance-wise (no token needed), but for staged > creates, the > >> > >> > > >> > >> > authorizer needs some extra info that only a token can carry. > With > >> > >> > > >> > >> > this context in mind, I would like to repurpose signing tokens as > >> > >> > > >> > >> > follows: every table load, create, register and staged create > would > >> > >> > > >> > >> > return a token carrying only the table context (operation, > catalog, > >> > >> > > >> > >> > table; locations as well for staged-creates) and the principal > hash, > >> > >> > > >> > >> > but not its privileges. Each sign request would still be > authorized > >> > >> > > >> > >> > individually: on the table for existing tables, or on the > namespace > >> > >> > > >> > >> > for staged-creates. Permission changes therefore take effect > >> > >> > > >> > >> > immediately (modulo the decision cache TTL). The token allows > >> > >> > > >> > >> > staged-creates to operate optimally, and ties each sign request > to its > >> > >> > > >> > >> > initial operation, which improves auditing. Also, since it > carries > >> > >> > > >> > >> > nothing secret anymore, it could be signed rather than encrypted, > >> > >> > > >> > >> > which is cheaper to verify. > >> > >> > > >> > >> > > >> > >> > > >> > >> > Batch sign: agreed, batch signing would help both models; but > note > >> > >> > > >> > >> > that once it lands in the spec, it still needs to get support at > least > >> > >> > > >> > >> > in the Java signer client, so it's not something we can count on > >> > >> > > >> > >> > today. > >> > >> > > >> > >> > > >> > >> > > >> > >> > Permissions: my rationale is that Polaris signs requests with > its own > >> > >> > > >> > >> > credentials, and I didn’t want this to be possible without an > explicit > >> > >> > > >> > >> > permission. But the catalog-level REMOTE_SIGNING_ENABLED flag > already > >> > >> > > >> > >> > lets admins opt in, so I'm OK to drop TABLE_REMOTE_SIGN and rely > on > >> > >> > > >> > >> > TABLE_READ_DATA / TABLE_WRITE_DATA, if everybody is on board. > >> > >> > > >> > >> > > >> > >> > > >> > >> > @Dmitri: > >> > >> > > >> > >> > > >> > >> > > >> > >> > The new token I'm proposing doesn’t need to expire for security > >> > >> > > >> > >> > reasons (it's not a security token), only for housekeeping sake > (allow > >> > >> > > >> > >> > key rotation, for instance). Therefore, I think one validity > window is > >> > >> > > >> > >> > enough, and one that is relatively generous is fine: I'd go with > 24 > >> > >> > > >> > >> > hours by default, configurable. > >> > >> > > >> > >> > > >> > >> > > >> > >> > Thanks, > >> > >> > > >> > >> > Alex > >> > >> > > >> > >> > > >> > >> > > >> > >> > On Sat, Nov 7, 2026 at 2:18 AM Dmitri Bourlatchkov via dev > >> > >> > > >> > >> > <[email protected]> wrote: > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > Hi Alex, > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > It's great to see progress on remote signing. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > I think the dual approach you proposed is reasonable. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > Re: staged creates: the long-lived signing token is similar to > a vended > >> > >> > > >> > >> > > credential. I believe it has the same problem that if the > staging operation > >> > >> > > >> > >> > > takes a very long time, the token (or credential) can expire > [1]. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > However, with signing tokens the Polaris administrator has > more control > >> > >> > > >> > >> > > over assigning the appropriate credential validity window, so > I think it is > >> > >> > > >> > >> > > a useful feature to have. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > It might be worth supporting different validity window > configurations and > >> > >> > > >> > >> > > different authorizable operations for normal table access and > staged create > >> > >> > > >> > >> > > requests. WDYT? > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > [1] > https://lists.apache.org/thread.html/by053ptrxrxqx9fm57q64bl3bv41vy46 > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > Cheers, > >> > >> > > >> > >> > > Dmitri. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > On Fri, Nov 6, 2026 at 12:04 PM Alex Dutra via dev < > [email protected]> > >> > >> > > >> > >> > > wrote: > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > > Hi Tommy, hi all, > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Your assistance in testing and evaluating this feature is > very welcome! > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Here are a few updates on my work: > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Since the old design doc [1] is mostly out of date, I'm > currently > >> > >> > > >> > >> > > > exploring two design choices for authorizing sign requests > that we > >> > >> > > >> > >> > > > briefly discussed, but never settled on: > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > 1. A signing token, issued when the table is loaded and > validated by > >> > >> > > >> > >> > > > the sign endpoint; this only works with Iceberg 1.12+. This > is similar > >> > >> > > >> > >> > > > to how Project Nessie does remote signing. In this mode, the > >> > >> > > >> > >> > > > authorization decision is expected to be fast, because it is > >> > >> > > >> > >> > > > pre-computed in the token. > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > 2. Per-request authorization, with or without a short-lived > decision > >> > >> > > >> > >> > > > cache; this works for any Iceberg client. In this mode, > authorization > >> > >> > > >> > >> > > > is done the normal way, which may incur in calls to the > database > >> > >> > > >> > >> > > > and/or to the external authorizer, if any. The decision > cache is a > >> > >> > > >> > >> > > > small enhancement where an authorization decision is cached > for 30 > >> > >> > > >> > >> > > > seconds by default. > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > To compare them, I created a new Gatling benchmark for > remote signing > >> > >> > > >> > >> > > > [2], and I'm collecting preliminary results by running it > locally > >> > >> > > >> > >> > > > against my working branch. A few early numbers (single > machine, > >> > >> > > >> > >> > > > Polaris capped at 4 CPUs, 1000 sign requests per table load, > so take > >> > >> > > >> > >> > > > them as indicative only), in max sign requests/s per node: > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > - JDBC: token 12.7k, per-request without cache 10.3k (-19%), > >> > >> > > >> > >> > > > per-request with cache 12.4k > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > - NoSQL: token 11.8k, per-request without cache 4.8k (-60%), > >> > >> > > >> > >> > > > per-request with cache 12.0k > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Using per-request authz without a decision cache is clearly > the worst > >> > >> > > >> > >> > > > scenario, as expected: the gap is mostly due to entity > resolution > >> > >> > > >> > >> > > > costs, and also cache deserialization for the NoSQL backend. > I also > >> > >> > > >> > >> > > > tested external principals (credential-mode=external) and > OPA: the > >> > >> > > >> > >> > > > numbers are similar or slightly worse due to repeated calls > to OPA. > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > However, per-request authz with a decision cache is roughly > on par > >> > >> > > >> > >> > > > with signing tokens. That's an interesting observation and > begs the > >> > >> > > >> > >> > > > question: which method should Polaris implement? Both? > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Each approach has pros and cons: > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > - Signing tokens: > >> > >> > > >> > >> > > > - Pros: works out of the box with staged table creations: > we capture > >> > >> > > >> > >> > > > the authz decision at load time > >> > >> > > >> > >> > > > - Cons: tokens are valid for X hours, and cannot be > revoked or > >> > >> > > >> > >> > > > refreshed (at least not for now - maybe later) > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > - Per-request authz with decision cache: > >> > >> > > >> > >> > > > - Pros: cache is short-lived (a few seconds) compared to > token lifetime > >> > >> > > >> > >> > > > - Cons: does not work well with staged table creations > (the sign > >> > >> > > >> > >> > > > request arrives before the table commit) > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > I am inclined to use a mix of both: per-request authz by > default, > >> > >> > > >> > >> > > > except for staged table creations where we'd use signing > tokens. > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > What are your thoughts? > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Thanks, > >> > >> > > >> > >> > > > Alex > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > [1]: > >> > >> > > >> > >> > > > > https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?usp=sharing > >> > >> > > >> > >> > > > [2]: https://github.com/apache/polaris-tools/pull/304 > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > On Mon, Nov 2, 2026 at 9:27 PM Tommy Bollhofer < > [email protected]> > >> > >> > > >> > >> > > > wrote: > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Hi Alex, > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Thank you for the update, and for linking the original > thread. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Great to hear that you're planning to revisit the previous > PR. We'd be > >> > >> > > >> > >> > > > happy to test an experimental build against our on-premises > FlashBlade > >> > >> > > >> > >> > > > S3-compatible storage and share our findings, including > compatibility > >> > >> > > >> > >> > > > across the various engines and versions we're currently > using. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Our client stack includes Trino with Iceberg 1.12, Spark > with the > >> > >> > > >> > >> > > > Iceberg 1.10 runtime, PyIceberg, and DuckDB. Given the > relatively short > >> > >> > > >> > >> > > > duration of our workloads, we should be able to begin > testing without > >> > >> > > >> > >> > > > requiring credential refresh support. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Please let us know when there's a build available to > evaluate. We'd be > >> > >> > > >> > >> > > > glad to help validate the implementation and provide > feedback. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Thanks again for your work on this! > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Best, > >> > >> > > >> > >> > > > > Tommy > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > ________________________________________ > >> > >> > > >> > >> > > > > From: Alex Dutra <[email protected]> > >> > >> > > >> > >> > > > > Sent: Tuesday, October 27, 2026 13:19 > >> > >> > > >> > >> > > > > To: [email protected] <[email protected]> > >> > >> > > >> > >> > > > > Cc: Tommy Bollhofer <[email protected]> > >> > >> > > >> > >> > > > > Subject: Re: [DISCUSS] Resume remote signing work > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Hi Tommy, > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Thanks for chiming in! Your reply got disconnected from > the thread > >> > >> > > >> > >> > > > though, so here is the link to my first message just in case: > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > https://lists.apache.org/thread.html/vk5zj7qtrztl36c9c1qt6d09lryjltdk > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > The details you provide are very valuable: since > FlashBlade doesn't > >> > >> > > >> > >> > > > support AssumeRole, remote signing becomes indeed a good > alternative. And > >> > >> > > >> > >> > > > since you can live for now without auth-credentials refresh, > it seems to me > >> > >> > > >> > >> > > > we have enough evidence that we should move forward and > provide a first, > >> > >> > > >> > >> > > > experimental implementation of remote signing. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > I will start preparing a revisited PR, built on top of the > old one. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Thanks, > >> > >> > > >> > >> > > > > Alex > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > On Thu, Oct 22, 2026 at 7:06 PM Tommy Bollhofer via dev < > >> > >> > > >> > >> > > > [email protected]> wrote: > >> > >> > > >> > >> > > > > Hi all, > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > +1 to resuming the remote signing work now, rather than > waiting for the > >> > >> > > >> > >> > > > > spec on refreshing auth data. Alex asked on the issue > (#32) for our use > >> > >> > > >> > >> > > > > case, so here it is. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > We run Polaris against an on-prem S3-compatible store (a > FlashBlade). > >> > >> > > >> > >> > > > > Its STS support is limited to AssumeRoleWithWebIdentity, > so Polaris > >> > >> > > >> > >> > > > > cannot vend credentials through its current AssumeRole > flow. As a > >> > >> > > >> > >> > > > > result we turn vending off and distribute static storage > keys to every > >> > >> > > >> > >> > > > > service in every environment: Polaris, Trino, Spark (table > maintenance), > >> > >> > > >> > >> > > > > and PyIceberg and DuckDB clients. That is a rotation and > blast-radius > >> > >> > > >> > >> > > > > burden that grows with each environment we add. Remote > signing would > >> > >> > > >> > >> > > > > remove the static keys from the clients, which is why we > would like to > >> > >> > > >> > >> > > > > see it move. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > On the auth-refresh gap: our workloads are mostly short > queries and > >> > >> > > >> > >> > > > > pipeline runs, and our longest jobs are Spark maintenance > operations > >> > >> > > >> > >> > > > > that take minutes, so as far as we can tell we could start > without > >> > >> > > >> > >> > > > > refresh support and pick it up incrementally, as Alex > suggested. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Client versions are mixed on our side (Trino on Iceberg > 1.12, Spark on > >> > >> > > >> > >> > > > > the Iceberg 1.10 runtime, PyIceberg and DuckDB with their > own Iceberg > >> > >> > > >> > >> > > > > implementations), so we expect client-side support to > vary. We would be > >> > >> > > >> > >> > > > > happy to test early builds against FlashBlade and report > which engines > >> > >> > > >> > >> > > > > and versions work. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > (On #32 we also described a smaller option, > AssumeRoleWithWebIdentity > >> > >> > > >> > >> > > > > as an alternative STS flow. Remote signing is the fuller > fix and the > >> > >> > > >> > >> > > > > one we would rather see.) > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Best, > >> > >> > > >> > >> > > > > Tommy > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > On 2026-10-02 15:00 UTC Alex Dutra wrote: > >> > >> > > >> > >> > > > > > Hi all, > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > Polaris will soon be updated to Iceberg 1.12. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > This release includes the new remote-signing-config > field within > >> > >> > > >> > >> > > > > > LoadTableResult, which addresses nearly all remaining > requirements for > >> > >> > > >> > >> > > > > > Polaris to complete its remote signing implementation, > abandoned a > >> > >> > > >> > >> > > > > > while ago due to impending spec changes. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > However, one detail remains: when we added the > remote-signing-config > >> > >> > > >> > >> > > > > > structure, refreshing the contained authentication or > authorization > >> > >> > > >> > >> > > > > > data was intentionally left out of scope. I plan to > start a separate > >> > >> > > >> > >> > > > > > discussion and spec change proposal with the Iceberg > community > >> > >> > > >> > >> > > > > > regarding this shortly. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > Despite this, the core remote signing functionality is > again viable in > >> > >> > > >> > >> > > > > > Polaris today, with Iceberg 1.12. The only missing piece > being that > >> > >> > > >> > >> > > > > > auth data cannot be refreshed yet, though this > capability could be > >> > >> > > >> > >> > > > > > added incrementally in a future update. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > With that in mind, I would like to ask the Polaris > community: > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > Should we resume the remote signing work where we left > [1]? Or would > >> > >> > > >> > >> > > > > > it be preferable to wait until the specification for > refreshing auth > >> > >> > > >> > >> > > > > > data is finalized? > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > For additional background, you can refer to the original > DISCUSS > >> > >> > > >> > >> > > > > > thread [2], but keep in mind that much of it is now > outdated. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > Thanks, > >> > >> > > >> > >> > > > > > Alex > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > [1]: https://github.com/apache/polaris/pull/3995 > >> > >> > > >> > >> > > > > > [2]: > >> > >> > > >> > >> > > > > https://lists.apache.org/thread.html/xpf60ko823c7mlg5bhnddq3btvdsq5s1 > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Tommy Bollhofer > >> > >> > > >> > >> > > > > Senior Data Engineer > >> > >> > > >> > >> > > > > Voloridge.com > >> > >> > > >> > >> > > > > [email protected] > >> > >> > > >> > >> > > > > 561.231.5770 <(561)%20231-5770> <(561)%20231-5770> > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > This email and any attachments are intended only for the > intended > >> > >> > > >> > >> > > > recipient(s) and may contain information that is privileged > and/or > >> > >> > > >> > >> > > > confidential. If you are not an intended recipient, or have > received this > >> > >> > > >> > >> > > > email in error, please notify the sender immediately and > permanently delete > >> > >> > > >> > >> > > > this email, its attachments, and any copies. Any disclosure, > copying, > >> > >> > > >> > >> > > > distribution, or reliance on the contents by anyone other > than an intended > >> > >> > > >> > >> > > > recipient is unauthorized. This email message (including any > attachments) > >> > >> > > >> > >> > > > is not to be construed as investment advice or the > solicitation of an offer > >> > >> > > >> > >> > > > to buy or sell interests in any securities. Any information > provided in > >> > >> > > >> > >> > > > this message must be considered in conjunction with the > appropriate > >> > >> > > >> > >> > > > disclosure documents. PAST PERFORMANCE IS NOT INDICATIVE OF > FUTURE RESULTS. > >> > >> > > >> > >> > > > > >> > >> > >
