Hi all,

+1 on #6028 from me. I think new Polaris list operations
should follow the token from the first response on the next
request, and the Iceberg complete-result promise stays where
it is, on the Iceberg operations (#5553 included).

One thing to watch in the impl. Right now
CatalogHandler.rejectIncompleteListing only rejects when
both pageToken and pageSize are missing and a next token
comes back [1]. That's really the Iceberg unsized rule. For
these new operations the first response is allowed to carry
a next-page token, and a missing or empty token both just
mean "first page". So if #5943 calls that helper, it'd 400
a first page that's actually valid under these rules.

Thanks,
Vignesh

[1]
https://github.com/apache/polaris/blob/67d10ae1629927fbac048cd4aa09a5906a1cd5dd
/runtime/service/src/main/java/org/apache/polaris
/service/catalog/common/CatalogHandler.java#L124

On Sat, 14 Nov 2026 at 13:02, Jean-Baptiste Onofré <[email protected]> wrote:

> Hi Rishi,
>
> I'm +1 on the approach. I don't think it conflicts with what we agreed in
> September. Fail-closed protects clients written against the Iceberg promise
> that a request with no pageToken gets the complete result. A new Polaris
> native operation makes no such promise, so "always follow the token" is the
> cleaner contract, and it bounds response size from day one. One shared
> definition is also better than a copy per operation.
>
> We had a similar discussion about pagination in the Directory Spec PR.
>
> I have a few comments (nothing blocking PR #6028):
> - As Prithvi noted in the September thread, Java Iceberg clients send an
> empty pageToken to start paginating. These operations always allow
> pagination, so I would state in the spec that an empty pageToken means the
> same as no pageToken. Then a client with Iceberg habits doesn't get a 400.
> - The storage configuration listings have filters (targetExists,
> referenced). It would be good to say that a client must send the same
> parameters on every page, apart from pageToken and pageSize. The spec could
> also say the server may reject a token that comes with different
> parameters.
> - LIST_PAGINATION_MAX_PAGE_SIZE is unlimited by default. If the new
> operations paginate only when an operator sets a maximum, a client that
> ignores the token works in every default deployment. Then it silently loses
> results the day someone sets a limit :) That's the September problem again,
> just moved :) I would suggest that these operations use a reasonable
> default page size even when no maximum is set, so clients use the token
> from the start. This belongs to the impl PRs after #5943, not in #6028.
> - #5533 showed that the generic tables CLI followed next-page-token only
> when --page-size was set. When the storage configuration operations land,
> the Python CLI should always follow the token.
>
> Regarding getApplicablePolicies, I would keep the Iceberg rules and fix
> #5311 under the current contract, the same as listPolicies. Putting one
> policy operation under different rules would be confusing, and it would
> change the contract for existing clients. If we ever want to move all the
> Polaris catalog listings (policies, generic tables, semantic models, ...)
> to these definitions, that should be its own discussion covering all of
> them.
>
> The Shares and Directory APIs already behave this way, so switching it to
> reference the new file could be a follow-up with no behavior change.
>
> Regards
> JB
>
> On Sat, Nov 14, 2026 at 2:18 AM Srinivas Rishindra <[email protected]
> >
> wrote:
>
> > Hi all,
> >
> > PR #6028 [1] adds spec/polaris-pagination.yaml, a small set of pagination
> > definitions for Polaris-specific list operations: a pageToken and a
> > pageSize
> >  query parameter, and a page token schema. Please give explicit sign-off
> > here before any operation starts using it, because the rules differ from
> > the Iceberg REST Catalog API. I'd like to settle that.
> >
> > The file states four rules:
> >
> >    1. Any response of an operation that uses these definitions may be
> >    paginated. A client must always check for a page token in the response
> > and,
> >    when one is present, send it unchanged as the pageToken query
> parameter
> >    to get the next page. The server omits the page token from the last
> > page. A
> >    client must treat a missing or null page token as the end of the
> > results.
> >    2. A client does not need to send pageToken or pageSize to start
> >    paginating. The server may paginate a response to a request that has
> >    neither.
> >    3. The server decides the page size. A page may hold fewer results
> than
> >    pageSize.
> >    4. A server that does not paginate returns all results in one
> response,
> >    with no page token.
> >
> > Scope: the rules apply only to operations that reference this file.
> Nothing
> > on main references it today. Operations that use the Iceberg REST Catalog
> > pagination definitions keep the Iceberg rules, including the Polaris
> > catalog listings that reuse them (policies, generic tables, semantic
> > models)
> > .
> >
> > How this relates to the September thread [2] and #5553 [3]: that thread
> > found that a configured LIST_PAGINATION_MAX_PAGE_SIZE truncated requests
> > with no pageToken, while the Iceberg spec promises such a request the
> > complete result. We agreed to reject those requests rather than truncate
> > them, and #5553 did that. The concern there was clients that rely on
> > the Iceberg
> > promise and never look for a token. These rules don't touch those
> > operations. For a new operation that uses this file, no such clients
> > exist: the
> > contract tells every client from day one to follow the token, so a first
> > page is never mistaken for the whole result. It also gives the server the
> > bound on response size that motivated the max page size in the first
> place.
> >
> > There is precedent: the Shares API spec on main (
> > spec/polaris-shares-api.yaml) already says that when pageSize is absent
> the
> > service's maximum is used, and that nextPageToken is present when more
> > results exist. This file writes that behavior down once for reuse.
> >
> > First intended users: the two list operations in the storage
> configuration
> > spec, PR #5943 [4] (listStorageConfigurations and listStorageMappings, in
> > the management API).
> >
> > One related question: getApplicablePolicies declares the Iceberg
> pagination
> > parameters but doesn't paginate yet (#5311 [5]). Should it keep the
> Iceberg
> > rules when that is fixed, like listPolicies next to it, or move to these
> > definitions? Moving it would change the contract of an existing
> operation,
> > so
> > I'd treat it as a separate decision either way.
> >
> > Please reply with +1 or any objections. If there are none in about 72
> > hours,
> >  I'll take that as agreement and ask for #6028 to be merged.
> >
> > [1] https://github.com/apache/polaris/pull/6028 [2]
> > https://lists.apache.org/thread/k81ptyktdbdf8gynncgk3o04mqt85zyk [3]
> > https://github.com/apache/polaris/pull/5553 [4]
> > https://github.com/apache/polaris/pull/5943 [5]
> > https://github.com/apache/polaris/issues/5311
> >
> > Thanks,
> > Rishi
> >
>

Reply via email to