laserninja opened a new pull request, #12241: URL: https://github.com/apache/gravitino/pull/12241
### What changes were proposed in this pull request? Adds `design-docs/iceberg-rest-server-side-scan-planning.md`, the design for completing the Iceberg REST two-step scan planning protocol in the Gravitino Iceberg REST server, as requested in [#12194](https://github.com/apache/gravitino/pull/12194#issuecomment-5102634079). The document covers: - **Background**: `POST .../plan` is implemented, `POST .../tasks` is not, so clients that gate on the advertised endpoint set (pyiceberg with `scan-planning-mode=server`) cannot use server-side planning at all, and a plan of a large table has to be returned in one unbounded response. - **Solution Investigations**: four token representations compared - always-inline (status quo), an in-memory plan-task store with random ids, a self-describing token (chosen), and asynchronous planning with server-side plan state - each with the concrete reason it was accepted or rejected, plus the cost of the chosen option stated in a table. - **Proposal**: - the new `POST .../tasks` interface (request fields, response schema, behavior, and the full error table), - the changed `POST .../plan` interface (old vs. new response, per-client migration impact), - the `plan-task` token: payload fields, encoding, validation, forward compatibility, and why the token is not a capability, - determinism: why the planned snapshot is pinned and why planned tasks are ordered totally, - interaction with the existing scan plan cache, - the internal dispatcher interfaces added, events and audit operation type, - federated catalogs, delete files and merge-on-read tables, configuration, user process, and implementation flow. - **Backward compatibility** and **Testing** summaries. - **Open questions** the implementation deliberately leaves to review: whether batching should require the scan plan cache to be enabled, whether 100 is the right default batch size, and whether tokens should carry an expiry. - **Task Breakdown**: what #12194 delivers, and the follow-ups that should be their own issues. ### Why are the changes needed? The implementation in #12194 introduces a new token type and changes what `POST .../plan` returns, and reviewers asked for the design to be written down and reviewed separately from the code. Fix: #11284 ### Does this PR introduce _any_ user-facing change? No. Documentation only; the design document describes changes proposed in #12194. ### How was this patch tested? No code change. The document was checked against the implementation in #12194: every interface, config default, error status and behavior it describes was verified against the code on that branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
