xiangfu0 opened a new pull request, #20009:
URL: https://github.com/apache/pinot/pull/20009
## Summary
Deleting a table (REST `DELETE /tables/{name}` or DDL `DROP TABLE`) force
deletes the table's non-running minion tasks through
`PinotHelixTaskResourceManager.deleteTask`. `DELETE
/tasks/task/{name}?forceDelete=true` uses the same method. If the Helix task
pipeline is still writing the job context at that moment, for example right
after the task was scheduled or its queue was stopped, Helix cannot remove
`/PROPERTYSTORE/TaskRebalancer/<job>` (ZooKeeper `NotEmpty`). It then throws
`HelixException: Failed to delete job ...` without a cause, and the API returns
500.
This change retries a forced `deleteJob` that throws `HelixException`, up to
3 attempts with 100 ms and 200 ms backoff. Retrying is safe because each step
of Helix's forced delete (job config, IS/EV, context) skips ZNodes that are
already gone, so a retry continues from where the last attempt failed. After
the last attempt, the original exception is rethrown, so the error message
stays the same. Non-forced deletes are not retried.
This race is behind the intermittent
`PinotTableRestletResourceTest.testTableTasksCleanupWithNonActiveTasks` failure
in "Pinot Unit Test Set 2", e.g.
https://github.com/apache/pinot/actions/runs/39855837737/job/120093687170.
## Testing
- New `PinotHelixTaskResourceManagerTest` cases: a forced delete succeeds
after one failure, gives up after the max attempts and rethrows the original
exception, and a non-forced delete is not retried.
- `./mvnw -pl pinot-controller -am
-Dtest=PinotTableRestletResourceTest,PinotHelixTaskResourceManagerTest
-Dsurefire.failIfNoSpecifiedTests=false test`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
