Hi All, I do not see any concerns with this change. If there are no fresh comments, I propose merging EOD today.
Cheers, Dmitri. On Tue, Sep 15, 2026 at 11:01 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Ayush, > > Your PR LGTM. > > In my view, the server should be able to truncate any response to a > manageable size and issue a continuation token. This is a rather basic > premise that enables servers to control overload. Sadly the IRC spec does > not allow that, but this deviation from the spec is quite reasonable > from my point of view. > > The proposed configuration options allow Polaris administrators to protect > their servers or elect to support simplistic clients (which cannot process > paginated responses) per the IRC spec, accepting the risk of server > overload. The choice is deployment-specific. > > I approved PR 5282 in GH. > > Cheers, > Dmitri. > > On Mon, Sep 14, 2026 at 9:52 PM Ayush Saxena <[email protected]> wrote: > >> Hi All, >> From the discussions at [1] which adds a configurable maximum at the >> pageSize specified by the client at server to avoid unbounded >> responses which can add pressure at N/w and on the server. The Iceberg >> REST specification treats a client's page-size as an upper bound, so a >> server is free to return fewer results >> >> During that discussion a spec concern came up which we would like the >> community's opinion on. >> >> The specification also says [2]: >> >> ``` >> Servers that support pagination must return all results in a single >> response with the value of next-page-token set to null if the query >> parameter pageToken is not set in the request. >> ``` >> >> So the two rules pull in different directions. A client that sends no >> pageToken at all is not asking for pagination, and the server is >> required to answer it completely. A server-side maximum, if it applies >> to such a request, truncates it and hands back a continuation token >> instead. >> >> That matters because of what a non-paginating client does with the >> response. A client that issues one unparameterized list request and >> returns whatever comes back has no reason to look at next-page-token, >> so it would silently report a partial listing as the complete one. >> Some old clients do exactly that; others send an empty pageToken and >> follow continuations, and those are unaffected either way. >> >> So the question is whether the maximum should apply to a request that >> did not ask for pagination. >> >> If it does apply, the protection covers every request, including the >> unbounded ones that motivated the change — but Polaris knowingly >> departs from the specification for clients that do not paginate, and >> those clients get paginated results rather than an error, which is the >> failure mode those old clients cannot detect. >> >> If it does not apply — i.e. we bound only requests that supply a >> pageToken, including an empty one — the spec contract is preserved >> exactly, and no client can be silently truncated. The cost is that the >> protection then reaches only clients that were already willing to >> paginate, while a client asking for an arbitrarily large single page >> is exactly the case it does not cover. >> >> As it stands the PR takes the first option, with two mitigations: the >> maximum defaults to unlimited, so it is entirely opt-in and the >> shipped behaviour is unchanged and spec-conforming; and the deviation >> is documented on the configuration itself so it is visible where an >> operator enables it. The reasoning being that an administrator knows >> their client population and can decide — leave it off where >> non-paginating clients exist, turn it on where all legitimate clients >> follow continuations. >> >> Let me know if it sounds good >> >> -Ayush >> >> >> [1] https://github.com/apache/polaris/pull/5282 >> [2] >> https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/open-api/rest-catalog-open-api.yaml#L2233-L2255 >> >
