nevzheng opened a new issue, #14390: URL: https://github.com/apache/gravitino/issues/14390
### What would you like to be improved? **Priority: P1** (scale in the parent epic; impact if hit, not scheduling) [IcebergCleanupManager.cleanupFiles](https://github.com/apache/gravitino/blob/bb634defdc5663a8124be5fca1c8d9e52ac096ac/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/cleanup/IcebergCleanupManager.java#L194-L226) reimplements table purge rather than following `CatalogUtil.dropTableData`, which sync purge uses. Wherever the two differ, async purge behaves differently from sync purge for the same table. Observed symptoms (each reproduced by a test in the parent epic): | Symptom | Effect | | ------- | ------ | | `gc.enabled=false` is never read. `dropTableData` skips data files in that case, because such tables (for example from Spark's `snapshot` procedure) point at another table's files. | Purging a snapshot table deletes the source table's data. | | [deleteDataFiles](https://github.com/apache/gravitino/blob/bb634defdc5663a8124be5fca1c8d9e52ac096ac/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/cleanup/IcebergCleanupManager.java#L337-L364) reads every manifest with `ManifestFiles.readPaths`, which throws `Cannot read a delete manifest with a ManifestReader` on delete manifests. That covers v2 position and equality deletes and v3 deletion vectors. | No table with row-level deletes can be async-purged. Each attempt deletes some data files first, then the job ends `FAILED`. | | `CatalogUtil.deleteFiles` swallows delete failures (#14333). | The job ends `SUCCEEDED` while files remain. | ### How should we improve? Keep the resumable, ordered deletion, but match `dropTableData`: honor `gc.enabled`, and read delete manifests with `ManifestFiles.readDeleteManifest`. Add parity tests that run both purges on the same table shapes (v1, v2 with deletes, v3 with deletion vectors, `gc.enabled=false`) and compare which files remain. Any fix should keep `dropTableData` semantics for `gc.enabled`. Filed for discussion and prioritization; no fix is scheduled as part of this report. Parent: the async purge audit epic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
