raghavyadav01 opened a new pull request, #20213:
URL: https://github.com/apache/pinot/pull/20213

   ## Summary
   
   `DistributedTaskLockManager` takes the table-level task-generation lock by 
creating an ephemeral ZK node. If ZooKeeper applies that create but the 
connection drops before the response arrives, the client retries the create, 
and the retry reports `NODE_EXISTS` for the node the first try created (or the 
call throws). The lock manager then reads this as "another controller holds the 
lock" and gives up. Nothing ever releases the node, so every later scheduling 
run for the table skips generation until the controller's ZK session ends.
   
   The fix: each acquire attempt writes a random `lockId` into the node. When 
`create` returns false or throws, the node is read back, and the lock is taken 
if it carries this attempt's id. The id is per attempt, not per controller, so 
another scheduling thread of the same controller, which shares the ZK session, 
never takes a node it did not create. Nodes written by older controllers lack 
the field and are never claimed.
   
   ## Testing
   
   **Unit tests** (`DistributedTaskLockManagerTest`, 11/11 pass):
   - New: the lock is taken when `create` reports `NODE_EXISTS` for its own 
node, and when it throws after creating it. Both fail without the fix.
   - New: a node written by another attempt is not taken.
   
   **Exhaustive interleaving check** 
(`DistributedTaskLockManagerInterleavingTest`): runs the real lock manager on 
two actors against a model of the ephemeral node and its ZK sessions. A 
scheduler allows one ZK operation at a time and enumerates, depth first:
   - every ordering of the two actors' operations through acquire, hold and 
release;
   - every outcome of each `create`: applied; applied but the retry reports 
`NODE_EXISTS`; applied but throws; not applied and throws;
   - zero or one session expiry at any point.
   
   Every run is checked for mutual exclusion (two holders while the first 
holder's session is alive) and for an orphan lock node once both actors finish. 
It covers two configurations, two threads of one controller and two 
controllers, at 95,400 schedules each. The fixed code passes all of them. The 
unfixed code fails within the first runs with `Orphan lock node left by 
controller1`. The check takes about 3 minutes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to