sheetalshah1007 opened a new pull request, #704:
URL: https://github.com/apache/atlas/pull/704
… partial-success responses, and runId-correlated admin audit APIs
Fix addEntity() duplicate check in EntityMutationResponse.
## Attachments
1. **Design
specification:**[ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-DESIGN.pdf](https://github.com/user-attachments/files/2296/ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-DESIGN.pdf)`
2. **Implementation notes:**
[ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md](https://github.com/user-attachments/files/30330473/ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md)
## Summary
This PR implements resilient bulk purge according to the **Resilient Bulk
Purge in Atlas Using Transaction Batching** design spec. It replaces the old,
single-transaction REST bulk purge with a pre-validated, worker-batch
processing model that is now shared by both REST and scheduled (cron) purges.
---
## What changes were proposed in this pull request?
### Key changes
- **Pre-validation** — REST GUIDs checked before writes (format, existence,
DELETED state, type). Invalid GUIDs return in `failedEntities`.
- **Request limit** — `atlas.purge.api.max.request.size` (default **1000**).
- **Shared worker path** — REST and cron use `PurgeBatchOrchestrator` /
`WorkItemManager` (`atlas.purge.worker.batch.size=100`,
`atlas.purge.workers.count=2`).
- **Batch isolation** — Each batch is its own transaction. Lock conflicts
retry at batch level (3 attempts, 500/1000/1500 ms). Other batch failures do
not roll back committed batches.
- **Partial success** — Committed batches stay committed when a later batch
fails.
- **Enriched response** — `failedEntities` + `PurgeSummary` (`runId`,
dependency counts, `executionFailed`).
- **EntityMutationResponse bugfix** — Replace O(n) list scans per
`addEntity()` with an O(1) GUID Set per operation; rebuild on
`setMutatedEntities()`.
- **Run-correlated audit** — Each run writes one **SUMMARY** row, **BATCH**
rows per worker batch, and failures to **`purgefailure.log`**, linked by
indexed **`runId`** and **`auditRowKind`**.
- **Admin audit APIs** — `/purgedEntities`, `/batches`, and `/details`
drill-down on summary and batch rows.
- **Cron alignment** — Same orchestration as REST; no monolithic AUTO_PURGE
audit row.
- **Parallel safety** — `accumulateDeletionCandidates()` returns GUIDs;
`DeleteHandlerV1` skips already-removed vertices (ATLAS-4766).
- **Schema** — `AUTO_PURGE`, `runId`, and `audit_row_kind` on
`__AtlasAuditEntry` (typedef patches 009/010).
### Compatibility notes
- `accumulateDeletionCandidates()` returns **`Set`** (not
`Set`).
- Ineligible GUIDs appear in **`failedEntities`**, not silently dropped.
- Completed purges return **HTTP 200** with outcome in the body (**400** for
empty input, size limit, auth, bad GUID length).
- REST and cron can overlap; cron-to-cron runs are serialized.
See `ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.md` for
flow, config, audit details, and design deviations.
---
## How was this patch tested?
| Area | Tests |
|--||
| End-to-end / cron audit | `PurgeTest` |
| Workers / retry | `PurgeBatchOrchestratorTest`, `PurgeBatchExecutorTest` |
| Graph | `AtlasEntityStoreV2Test`, `DeleteHandlerV1Test` |
| REST | `AdminResourceTest` |
| Client | `AtlasClientV2Test` |
---
## Performance testing
**Load:** 1,000 `hive_table` entities (2 columns each) — **5,000 entities**
total (tables + columns + storage-desc + ddl ).
| Config | Avg purge_ms | Time | vs pre-5317 |
||--|--|-|
| Pre-5317 (single transaction) | 273,318 | 4m 33s | baseline |
| Post-5317, `workers.count=2` | ~58,541 | ~59s | ~79% faster (~4.7×) |
| Post-5317, `workers.count=1` | ~109,471 | ~1m 49s | ~60% faster (~2.5×) |
### Regarding `atlas.purge.workers.count`
**Default = 2** — Cron purge historically used 2 workers; the new design
keeps that. One batch can commit while another runs — solid speed-up without
overwhelming the database.
| Setting | Behavior | When to use |
|-|--|-|
| **2** (default) | Two parallel batches | Regular large purges; best
balance of speed and load on this test |
| **1** | Sequential (one batch at a time) | Prefer legacy-like behavior;
still ~60% faster than pre-5317 |
| **> 2** (e.g. 5) | More parallel batches | Only after cluster testing —
may not scale linearly |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
