raghavyadav01 opened a new pull request, #20213: URL: https://github.com/apache/pinot/pull/20213
## Summary `DistributedTaskLockManager` takes the table-level task-generation lock by creating an ephemeral ZK node. If ZooKeeper applies that create but the connection drops before the response arrives, the client retries the create, and the retry reports `NODE_EXISTS` for the node the first try created (or the call throws). The lock manager then reads this as "another controller holds the lock" and gives up. Nothing ever releases the node, so every later scheduling run for the table skips generation until the controller's ZK session ends. The fix: each acquire attempt writes a random `lockId` into the node. When `create` returns false or throws, the node is read back, and the lock is taken if it carries this attempt's id. The id is per attempt, not per controller, so another scheduling thread of the same controller, which shares the ZK session, never takes a node it did not create. Nodes written by older controllers lack the field and are never claimed. ## Testing **Unit tests** (`DistributedTaskLockManagerTest`, 11/11 pass): - New: the lock is taken when `create` reports `NODE_EXISTS` for its own node, and when it throws after creating it. Both fail without the fix. - New: a node written by another attempt is not taken. **Exhaustive interleaving check** (`DistributedTaskLockManagerInterleavingTest`): runs the real lock manager on two actors against a model of the ephemeral node and its ZK sessions. A scheduler allows one ZK operation at a time and enumerates, depth first: - every ordering of the two actors' operations through acquire, hold and release; - every outcome of each `create`: applied; applied but the retry reports `NODE_EXISTS`; applied but throws; not applied and throws; - zero or one session expiry at any point. Every run is checked for mutual exclusion (two holders while the first holder's session is alive) and for an orphan lock node once both actors finish. It covers two configurations, two threads of one controller and two controllers, at 95,400 schedules each. The fixed code passes all of them. The unfixed code fails within the first runs with `Orphan lock node left by controller1`. The check takes about 3 minutes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
