Hi All, >From the discussions at [1] which adds a configurable maximum at the pageSize specified by the client at server to avoid unbounded responses which can add pressure at N/w and on the server. The Iceberg REST specification treats a client's page-size as an upper bound, so a server is free to return fewer results
During that discussion a spec concern came up which we would like the community's opinion on. The specification also says [2]: ``` Servers that support pagination must return all results in a single response with the value of next-page-token set to null if the query parameter pageToken is not set in the request. ``` So the two rules pull in different directions. A client that sends no pageToken at all is not asking for pagination, and the server is required to answer it completely. A server-side maximum, if it applies to such a request, truncates it and hands back a continuation token instead. That matters because of what a non-paginating client does with the response. A client that issues one unparameterized list request and returns whatever comes back has no reason to look at next-page-token, so it would silently report a partial listing as the complete one. Some old clients do exactly that; others send an empty pageToken and follow continuations, and those are unaffected either way. So the question is whether the maximum should apply to a request that did not ask for pagination. If it does apply, the protection covers every request, including the unbounded ones that motivated the change — but Polaris knowingly departs from the specification for clients that do not paginate, and those clients get paginated results rather than an error, which is the failure mode those old clients cannot detect. If it does not apply — i.e. we bound only requests that supply a pageToken, including an empty one — the spec contract is preserved exactly, and no client can be silently truncated. The cost is that the protection then reaches only clients that were already willing to paginate, while a client asking for an arbitrarily large single page is exactly the case it does not cover. As it stands the PR takes the first option, with two mitigations: the maximum defaults to unlimited, so it is entirely opt-in and the shipped behaviour is unchanged and spec-conforming; and the deviation is documented on the configuration itself so it is visible where an operator enables it. The reasoning being that an administrator knows their client population and can decide — leave it off where non-paginating clients exist, turn it on where all legitimate clients follow continuations. Let me know if it sounds good -Ayush [1] https://github.com/apache/polaris/pull/5282 [2] https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/open-api/rest-catalog-open-api.yaml#L2233-L2255
