sheetalshah1007 opened a new pull request, #704: URL: https://github.com/apache/atlas/pull/704
… partial-success responses, and runId-correlated admin audit APIs Fix addEntity() duplicate check in EntityMutationResponse. ## Attachments 1. **Design specification:**[ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-DESIGN.pdf](https://github.com/user-attachments/files/29999296/ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-DESIGN.pdf)` 2. **Implementation notes:** [ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md](https://github.com/user-attachments/files/30330473/ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md) ## Summary This PR implements resilient bulk purge according to the **Resilient Bulk Purge in Atlas Using Transaction Batching** design spec. It replaces the old, single-transaction REST bulk purge with a pre-validated, worker-batch processing model that is now shared by both REST and scheduled (cron) purges. --- ## What changes were proposed in this pull request? ### Key changes - **Pre-validation** — REST GUIDs checked before writes (format, existence, DELETED state, type). Invalid GUIDs return in `failedEntities`. - **Request limit** — `atlas.purge.api.max.request.size` (default **1000**). - **Shared worker path** — REST and cron use `PurgeBatchOrchestrator` / `WorkItemManager` (`atlas.purge.worker.batch.size=100`, `atlas.purge.workers.count=2`). - **Batch isolation** — Each batch is its own transaction. Lock conflicts retry at batch level (3 attempts, 500/1000/1500 ms). Other batch failures do not roll back committed batches. - **Partial success** — Committed batches stay committed when a later batch fails. - **Enriched response** — `failedEntities` + `PurgeSummary` (`runId`, dependency counts, `executionFailed`). - **EntityMutationResponse bugfix** — Replace O(n) list scans per `addEntity()` with an O(1) GUID Set per operation; rebuild on `setMutatedEntities()`. - **Run-correlated audit** — Each run writes one **SUMMARY** row, **BATCH** rows per worker batch, and failures to **`purgefailure.log`**, linked by indexed **`runId`** and **`auditRowKind`**. - **Admin audit APIs** — `/purgedEntities`, `/batches`, and `/details` drill-down on summary and batch rows. - **Cron alignment** — Same orchestration as REST; no monolithic AUTO_PURGE audit row. - **Parallel safety** — `accumulateDeletionCandidates()` returns GUIDs; `DeleteHandlerV1` skips already-removed vertices (ATLAS-4766). - **Schema** — `AUTO_PURGE`, `runId`, and `audit_row_kind` on `__AtlasAuditEntry` (typedef patches 009/010). ### Compatibility notes - `accumulateDeletionCandidates()` returns **`Set<String>`** (not `Set<AtlasVertex>`). - Ineligible GUIDs appear in **`failedEntities`**, not silently dropped. - Completed purges return **HTTP 200** with outcome in the body (**400** for empty input, size limit, auth, bad GUID length). - REST and cron can overlap; cron-to-cron runs are serialized. See `ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md` for flow, config, audit details, and design deviations. --- ## How was this patch tested? | Area | Tests | |------|--------| | End-to-end / cron audit | `PurgeTest` | | Workers / retry | `PurgeBatchOrchestratorTest`, `PurgeBatchExecutorTest` | | Graph | `AtlasEntityStoreV2Test`, `DeleteHandlerV1Test` | | REST | `AdminResourceTest` | | Client | `AtlasClientV2Test` | --- ## Performance testing **Load:** 1,000 `hive_table` entities (2 columns each) — **5,000 entities** total (tables + columns + storage-desc + ddl ). | Config | Avg purge_ms | Time | vs pre-5317 | |--------|--------------|------|-------------| | Pre-5317 (single transaction) | 273,318 | 4m 33s | baseline | | Post-5317, `workers.count=2` | ~58,541 | ~59s | ~79% faster (~4.7×) | | Post-5317, `workers.count=1` | ~109,471 | ~1m 49s | ~60% faster (~2.5×) | ### Regarding `atlas.purge.workers.count` **Default = 2** — Cron purge historically used 2 workers; the new design keeps that. One batch can commit while another runs — solid speed-up without overwhelming the database. | Setting | Behavior | When to use | |---------|----------|-------------| | **2** (default) | Two parallel batches | Regular large purges; best balance of speed and load on this test | | **1** | Sequential (one batch at a time) | Prefer legacy-like behavior; still ~60% faster than pre-5317 | | **> 2** (e.g. 5) | More parallel batches | Only after cluster testing — may not scale linearly | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
