markhoerth opened a new issue, #14453: URL: https://github.com/apache/gravitino/issues/14453
### What would you like to be improved? None of the list endpoints of the Gravitino REST API (`/api/metalakes/...`) supports pagination. Every list request returns the complete set in one response. Large installations need pagination: a schema with tens of thousands of tables, a metalake with thousands of users, roles or tags. The Iceberg REST and Lance REST services already page their list responses (`pageToken`/`pageSize`, #10669, #11134). The main API does not. Each list handler loads the whole list, filters it for authorization in memory and returns it as one `EntityListResponse`. See `listTables` in `server/src/main/java/org/apache/gravitino/server/web/rest/TableOperations.java` (lines 88-110 at bb634defd). The API interfaces return whole arrays, for example `NameIdentifier[] listTables(Namespace)` in `api/src/main/java/org/apache/gravitino/rel/TableCatalog.java:54`, so neither the server nor the clients can ask for less. The list endpoints affected are listMetalakes, listCatalogs, listSchemas, listTables, listViews, listFunctions, listFilesets, listFiles, listModels, listModelVersions, listTopics, listPartitionNames, listUsers, listGroups, listRoles, listTags, listPolicies, listJobTemplates, listJobs, listSemanticModels, listMetrics, listStatistics and listPartitionStatistics. The association lists are affected too: tags, policies and roles for a metadata object, and metadata objects or policies for a tag. ### How should we improve? Requirements: 1. Every list endpoint under `/api` accepts `pageToken` and `pageSize` query parameters, named as in the Iceberg REST service, and returns `nextPageToken` when more results remain. The token is opaque to the client. 2. A request without `pageSize` behaves as today and returns the full list, so existing clients keep working. A server setting caps the largest page size the server will return. 3. Results come back in a stable order, so that walking the pages returns every entity exactly once while nothing is created or dropped. 4. Authorization filtering happens before the page is cut. A page holds up to `pageSize` entities the caller may see; it is never short or empty because hidden entities were dropped from it, and `nextPageToken` is absent only when nothing visible remains. 5. The `details=true` variants page the same way. 6. A malformed or expired `pageToken`, or a `pageSize` below 1, is rejected with a 400 that names the parameter. 7. The Java and Python clients expose paged listing alongside the existing methods. 8. Tests cover a walk across several pages and a page boundary falling on entities the caller cannot see. Paging under concurrent creates and drops is tested too. Related: #13631 (paginated listing and counting in EntityStore) and its epic #13621, #11833 (paginated listUsers/listGroups in core), #4576 (pagination for list tags), #11775 (N+1 cost of authorized table listing), and #10669 and #11134 (pagination on the Iceberg REST list endpoints). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
