roryqi opened a new issue, #11265: URL: https://github.com/apache/gravitino/issues/11265
### Describe the feature When a client drops an Iceberg table with `purgeRequested=true` through the Iceberg REST catalog, Gravitino deletes all data and metadata files inline on the request thread. This proposes an **opt-in asynchronous purge**: the drop returns quickly after durably recording intent, and a background worker performs the file deletion idempotently. ### Motivation - Inline purge of a large table can block the REST request for a long time and risk client timeouts. - A crash mid-deletion can leave files partially removed with no record to resume. A durable, async purge decouples client-facing drop latency from the cost of deleting files, and survives server restarts. ### Describe the solution - Persist purge jobs in a new `iceberg_cleanup_job` table with a small state machine (`PENDING -> RUNNING -> SUCCEEDED/FAILED`), claimed atomically by workers via a heartbeat lease. - On `dropTable` with the async header, enqueue a job and write a tombstone instead of deleting inline; reject create/register of the same table while a purge is pending. - A server-wide worker pool polls for free capacity, deletes files in batches, retries up to a cap, and prunes terminal rows past a retention window. - Synchronous deletion stays the default; async is opt-in. Delivered as three stacked PRs: (1) persistence layer, (2) purge manager engine, (3) REST integration. This is Phase 1; the observability/cleanups endpoint is deferred. ### Additional context _None._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
