aboueleyes opened a new issue, #17755: URL: https://github.com/apache/iceberg/issues/17755
### Apache Iceberg version 1.11.0 (latest release) ### Query engine None ### Please describe the bug 🐞 `RESTSessionCatalog` keeps requesting pages until a response has a null `next-page-token`. If a REST catalog returns a token that the client has already used, the listing never terminates. Each repeated response is also added to the result builder, so memory use and request volume continue growing while the call is blocked. I reproduced this against Iceberg 1.10.0 with a small HTTP stub that always returns the same token. The same loop structure is present in 1.11.0 and on `main` in these methods: - `listTables` - `listNamespaces` - `listViews` Current source: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java ### Measured result The stub returns this response for every namespace listing request: ```json {"namespaces":[["demo"]],"next-page-token":"stuck"} ``` Results from the same test: ```text control, next-page-token: null terminated after 1 request repeated token 51,043 requests in 5 seconds, still running ``` Every individual HTTP request completes successfully, so the connection and socket timeouts do not bound the overall listing operation. ### Expected behavior The client should fail with a clear error when pagination does not make progress, rather than making unlimited requests and accumulating repeated results. Returning a partial listing would be unsafe because callers could treat missing namespaces, tables, or views as a complete result. ### REST contract The REST specification requires the last response to have a null `next-page-token`. A server that repeats a token is therefore violating the contract: https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L2376 This report is about limiting the effect of that invalid response on the client. It is not claiming that repeated tokens are valid server behavior. ### Catalog implementations checked I used Codex to check the current implementations and pagination tests for Apache Polaris, Project Nessie, Lakekeeper, Unity Catalog, and Iceberg's REST fixture. None intentionally reuses a page token: - Polaris advances its listing position and returns null on the final page. - Nessie includes its underlying paging token only when more entries exist. - Lakekeeper advances through storage tokens and returns no token when exhausted. - Unity Catalog currently consumes its internal pages before returning a complete Iceberg listing. - Iceberg's REST fixture uses increasing numeric offsets and returns null at the end. It didn't find a released catalog that produces the repeated-token behavior. A catalog defect or an intermediary returning a stale response could still trigger it. ### Reproduction outline 1. Start an HTTP server that returns a normal `/v1/config` response. 2. Return one namespace and the same non-null `next-page-token` from every `/v1/namespaces` request. 3. Initialize `RESTCatalog` against the server. 4. Call `listNamespaces()`. 5. Observe that the call does not return and requests continue until the process is stopped. The healthy control changes only `next-page-token` to null and terminates after one request. ### Possible fix Remember tokens already used by each listing call and reject a token when it appears again. Tracking all tokens is necessary because comparing only with the previous token does not detect a server that alternates between two or more tokens. A page limit would also bound the loop, but it would impose an arbitrary maximum on valid catalogs. Before preparing a PR, I would like confirmation that failing on a repeated token is the preferred behavior. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
