sheetalshah1007 opened a new pull request, #702: URL: https://github.com/apache/atlas/pull/702
…ses, worker batches, and audit APIs ## Attachments 1. **Design specification:**[ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-DESIGN.pdf](https://github.com/user-attachments/files/29999296/ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-DESIGN.pdf)` 2. **Implementation notes:** [ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md](https://github.com/user-attachments/files/30251673/ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md) --- ## What changes were proposed in this pull request? This PR implements resilient bulk purge per the attached design specification (*Resilient Bulk Purge in Atlas Using Transaction Batching*). It replaces the old single-transaction REST bulk purge with pre-validated, worker-batch processing shared by REST and scheduled (cron) purge. ### Key changes - **Pre-validation before writes** — REST purge GUIDs are checked for UUID format, graph existence, DELETED state, and registered type before any mutation. Invalid GUIDs are returned in `failedEntities` with `errorCode` / `errorMessage`. - **Request size limit** — New config `atlas.purge.api.max.request.size` (default 1000) caps GUIDs per REST purge request. - **Shared worker-batch path** — REST and cron both use `PurgeBatchOrchestrator` / `WorkItemManager` with `atlas.purge.worker.batch.size` (default 100) and `atlas.purge.workers.count` (default 2). - **Isolated batch transactions** — Each worker batch runs in its own graph transaction. Lock conflicts retry at batch level; other batch failures roll back only that batch and processing continues. - **Partial success** — Earlier committed batches are kept when a later batch fails permanently. - **Enriched response** — `EntityMutationResponse` adds `failedEntities` and `PurgeSummary` (including `purgedDependenciesCount` / `failedDependenciesCount`). - **Per-batch audit** — Successful batches write bounded PURGE / AUTO_PURGE audit entries (up to 10 sample GUIDs), not one large entry per REST/cron call. - **Cron alignment** — Scheduled purge keeps its existing worker-batch model; REST is aligned to that path with shared orchestration, enriched accounting, batch retry, and shutdown reconciliation. - **Parallel-batch hardening** — `DeleteHandlerV1` tolerates vertices/edges already removed by another worker (ATLAS-4766). `accumulateDeletionCandidates()` returns `Set<String>` (GUIDs) for worker-safe enqueueing. - **Schema** — `AUTO_PURGE` added to `atlas_operation` in `0010-base_model.json` for scheduled purge audit on fresh installs. ### Compatibility and behavior notes - `accumulateDeletionCandidates()` now returns `Set<String>` (GUIDs) instead of `Set<AtlasVertex>`. Custom `AtlasEntityStore` implementations should update their method signature; Apache Atlas ships only `AtlasEntityStoreV2`. - Missing or ineligible GUIDs appear in `failedEntities` on REST (no longer silently ignored). - Completed purge responses return HTTP 200 with outcome in the body — read `summary` and `failedEntities` for the result (implementation plan Section 10). - Large purges produce multiple audit entries (one per successful worker batch). For full flow, design deviations, configuration, and review focus areas, see the attached implementation plan. --- ## How was this patch tested? Unit and module tests were added or extended in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
