Hi Ayush, Your PR LGTM.
In my view, the server should be able to truncate any response to a manageable size and issue a continuation token. This is a rather basic premise that enables servers to control overload. Sadly the IRC spec does not allow that, but this deviation from the spec is quite reasonable from my point of view. The proposed configuration options allow Polaris administrators to protect their servers or elect to support simplistic clients (which cannot process paginated responses) per the IRC spec, accepting the risk of server overload. The choice is deployment-specific. I approved PR 5282 in GH. Cheers, Dmitri. On Mon, Sep 14, 2026 at 9:52 PM Ayush Saxena <[email protected]> wrote: > Hi All, > From the discussions at [1] which adds a configurable maximum at the > pageSize specified by the client at server to avoid unbounded > responses which can add pressure at N/w and on the server. The Iceberg > REST specification treats a client's page-size as an upper bound, so a > server is free to return fewer results > > During that discussion a spec concern came up which we would like the > community's opinion on. > > The specification also says [2]: > > ``` > Servers that support pagination must return all results in a single > response with the value of next-page-token set to null if the query > parameter pageToken is not set in the request. > ``` > > So the two rules pull in different directions. A client that sends no > pageToken at all is not asking for pagination, and the server is > required to answer it completely. A server-side maximum, if it applies > to such a request, truncates it and hands back a continuation token > instead. > > That matters because of what a non-paginating client does with the > response. A client that issues one unparameterized list request and > returns whatever comes back has no reason to look at next-page-token, > so it would silently report a partial listing as the complete one. > Some old clients do exactly that; others send an empty pageToken and > follow continuations, and those are unaffected either way. > > So the question is whether the maximum should apply to a request that > did not ask for pagination. > > If it does apply, the protection covers every request, including the > unbounded ones that motivated the change — but Polaris knowingly > departs from the specification for clients that do not paginate, and > those clients get paginated results rather than an error, which is the > failure mode those old clients cannot detect. > > If it does not apply — i.e. we bound only requests that supply a > pageToken, including an empty one — the spec contract is preserved > exactly, and no client can be silently truncated. The cost is that the > protection then reaches only clients that were already willing to > paginate, while a client asking for an arbitrarily large single page > is exactly the case it does not cover. > > As it stands the PR takes the first option, with two mitigations: the > maximum defaults to unlimited, so it is entirely opt-in and the > shipped behaviour is unchanged and spec-conforming; and the deviation > is documented on the configuration itself so it is visible where an > operator enables it. The reasoning being that an administrator knows > their client population and can decide — leave it off where > non-paginating clients exist, turn it on where all legitimate clients > follow continuations. > > Let me know if it sounds good > > -Ayush > > > [1] https://github.com/apache/polaris/pull/5282 > [2] > https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/open-api/rest-catalog-open-api.yaml#L2233-L2255 >
