Thanks Kishore. I tested this patch on tag 0.7.1 because we are using 0.7.1. The code is same in master branch though.
Where do you want this patch to land? On Wed, May 25, 2016 at 8:35 PM, kishore g <g.kish...@gmail.com> wrote: > sorry I missed this. It looked good to me. Will merge it. > > On Wed, May 25, 2016 at 7:41 PM, Neutron sharc <neutronsh...@gmail.com> > wrote: > >> Hi Kishore, Kanak, any updates? >> >> On Thu, May 19, 2016 at 4:13 PM, kishore g <g.kish...@gmail.com> wrote: >> > Thanks Shawn. Will review it tonight. Kanak, It will be great if you can >> > take a look at it as well. >> > >> > On Thu, May 19, 2016 at 3:45 PM, Neutron sharc <neutronsh...@gmail.com> >> > wrote: >> > >> >> Hi Helix team, >> >> >> >> I uploaded a PR to fix this bug: >> https://github.com/apache/helix/pull/44 >> >> >> >> Thanks. >> >> >> >> On Wed, May 18, 2016 at 11:01 PM, Neutron sharc <neutronsh...@gmail.com >> > >> >> wrote: >> >> > Hi Kanak, >> >> > >> >> > The same problem with zk helix lock re-appears. I found some clues >> >> > about the potential bug. This potential bug causes all threads >> >> > competing for a same zk helix lock to be blocked. >> >> > >> >> > In my test there are two java threads blocked when trying to grab zk >> >> > lock (thread 15 and thread 19) >> >> > >> >> > Here are related logs before the threads are blocked (inlined with my >> >> comments) >> >> > >> >> > [INFO 2016-05-18 22:19:54,057 com.hcd.hcdadmin.M1Rebalancer:70] >> >> > rebalancer thread 15 before zklock >> >> > => T15 enters >> >> > >> >> > [DEBUG 2016-05-18 22:19:54,069 org.apache.helix.lock.zk.WriteLock:193] >> >> > Created id: >> /shawn1/LOCKS/RESOURCE_Pool0/x-72233245264911661-0000000078 >> >> > => T15 creates znode, T15 is the smallest so it owns lock >> >> > >> >> > [INFO 2016-05-18 22:19:54,071 com.hcd.hcdadmin.M1Rebalancer:70] >> >> > rebalancer thread 19 before zklock >> >> > => T19 enters >> >> > >> >> > [INFO 2016-05-18 22:19:54,071 com.hcd.hcdadmin.M1Rebalancer:72] >> >> > rebalancer thread 15 start computing for controller host1_admin >> >> > => T15 performs its work >> >> > >> >> > [DEBUG 2016-05-18 22:19:54,080 org.apache.helix.lock.zk.WriteLock:193] >> >> > Created id: >> /shawn1/LOCKS/RESOURCE_Pool0/x-72233245264911662-0000000079 >> >> > => T19 creates its znode >> >> > >> >> > [DEBUG 2016-05-18 22:19:54,081 org.apache.helix.lock.zk.WriteLock:233] >> >> > watching less than me node: >> >> > /shawn1/LOCKS/RESOURCE_Pool0/x-72233245264911661-0000000078 >> >> > => T19 found its predecessor to wait for, which is T15 >> >> > >> >> > [WARN 2016-05-18 22:19:54,084 org.apache.helix.lock.zk.WriteLock:239] >> >> > Could not find the stats for less than me: >> >> > /shawn1/LOCKS/RESOURCE_Pool0/x-72233245264911661-0000000078 >> >> > => T19 calls zookeeper.exist() to register a watcher on T15, but >> >> > T15 has called unlock() to delete the znode at the same moment. So >> >> > T19 continues to check while(id==null) loop. Because T19 id is not >> >> > null now, T19's LockZooKeeperOperation.execute() returns false. T19 >> >> > will block at wait(), hoping somebody else will notify it. But since >> >> > T19 is currently the smallest so nobody else can grab the lock and >> >> > wait up T19; T19 blocks, and every subsequent caller also blocks. >> >> > >> >> > The code that leads to the problem is here: >> >> > >> >> >> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/lock/zk/WriteLock.java#L238 >> >> > >> >> > One possible fix is to just set id to null at line 240 and let while() >> >> > loop to retry. >> >> > >> >> >> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/lock/zk/WriteLock.java#L240 >> >> > >> >> > >> >> > [INFO 2016-05-18 22:19:54,092 com.hcd.hcdadmin.M1Rebalancer:125] >> >> > rebalancer thread 15 released zklock and returned >> >> > => T15 has deleted znode a short while ago and returns from this >> >> method >> >> > >> >> > >> >> > [INFO 2016-05-18 22:19:54,179 com.hcd.hcdadmin.M1Rebalancer:70] >> >> > rebalancer thread 15 before zklock >> >> > => T15 calls this method again, >> >> > >> >> > [DEBUG 2016-05-18 22:19:54,191 org.apache.helix.lock.zk.WriteLock:193] >> >> > Created id: >> /shawn1/LOCKS/RESOURCE_Pool0/x-72233245264911663-0000000080 >> >> > => T15 creates znode >> >> > >> >> > [DEBUG 2016-05-18 22:19:54,193 org.apache.helix.lock.zk.WriteLock:233] >> >> > watching less than me node: >> >> > /shawn1/LOCKS/RESOURCE_Pool0/x-72233245264911662-0000000079 >> >> > >> >> > => T15 found T19 to be smallest so it waits for T19. Nobody will >> >> > wake up T19, so T15 is also blocked. >> >> > >> >> > >> >> > >> >> > >> >> > Any comments appreciated. Thanks. >> >> > >> >> > >> >> > -Neutronsharc >> >> > >> >> > >> >> > >> >> > On Sat, May 14, 2016 at 5:20 PM, Neutron sharc < >> neutronsh...@gmail.com> >> >> wrote: >> >> >> We increased the max connections allowed per client at zk server >> side. >> >> >> The problem is gone now. >> >> >> >> >> >> On Tue, May 10, 2016 at 2:50 PM, Neutron sharc < >> neutronsh...@gmail.com> >> >> wrote: >> >> >>> Hi Kanak, thanks for reply. >> >> >>> >> >> >>> The problem is gone if we set a constraint of 1 on >> "STATE_TRANSITION" >> >> >>> for the resource. If we allow multiple state transitions to be >> >> >>> executed in the resource, then this zklock problem occurs. >> >> >>> >> >> >>> btw, we run multiple participants in a same jvm in our test. In >> >> >>> other words, there are multiple java threads in a same jvm competing >> >> >>> for zklock. >> >> >>> >> >> >>> We haven't profiled the ZKHelixLock._listener.lockAcquired() since >> we >> >> >>> bypassed this problem using constraint. Will revisit it later. >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> On Mon, May 9, 2016 at 8:28 PM, Kanak Biscuitwala < >> kana...@hotmail.com> >> >> wrote: >> >> >>>> Hi, >> >> >>>> >> >> >>>> ZkHelixLock is a thin wrapper around the ZooKeeper WriteLock recipe >> >> (which was last changed over 5 years ago). Though we haven't extensively >> >> tested it in production, but we haven't seen it fail to return as >> described. >> >> >>>> >> >> >>>> Do you know if ZKHelixLock._listener.lockAcquired() is ever called? >> >> >>>> >> >> >>>> Feel free to examine the code here: >> >> >> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/lock/zk/ZKHelixLock.java >> >> >>>> >> >> >>>>> From: neutronsh...@gmail.com >> >> >>>>> Date: Mon, 9 May 2016 14:26:43 -0700 >> >> >>>>> Subject: calling ZKHelixLock from state machine transition >> >> >>>>> To: dev@helix.apache.org >> >> >>>>> >> >> >>>>> Hi Helix team, >> >> >>>>> >> >> >>>>> We observed an issue at state machine transition handle: >> >> >>>>> >> >> >>>>> // statemodel.java: >> >> >>>>> >> >> >>>>> public void offlineToSlave(Message message, NotificationContext >> >> context) { >> >> >>>>> >> >> >>>>> // do work to start a local shard >> >> >>>>> >> >> >>>>> // we want to save the new shard info to resource config >> >> >>>>> >> >> >>>>> >> >> >>>>> ZKHelixLock zklock = new ZKHelixLock(clusterId, resource, >> >> zkclient); >> >> >>>>> try { >> >> >>>>> zklock.lock(); // ==> will be blocked here >> >> >>>>> >> >> >>>>> ZNRecord record = zkclient.readData(scope.getZkPath(), true); >> >> >>>>> update record fields; >> >> >>>>> zkclient.writeData(scope.getZkPath(), record); >> >> >>>>> } finally { >> >> >>>>> zklock.unlock(); >> >> >>>>> } >> >> >>>>> } >> >> >>>>> >> >> >>>>> After several invocation of this method, zklock.lock() method >> >> doesn't >> >> >>>>> return (so the lock is not acquired). State machine threads >> become >> >> >>>>> blocked. >> >> >>>>> >> >> >>>>> At zk path "<cluster>/LOCKS/RESOURCE_resource" I see several >> znodes >> >> >>>>> there, representing outstanding lock requests. >> >> >>>>> >> >> >>>>> Are there any special care we should be aware of about zk lock ? >> >> Thanks. >> >> >>>>> >> >> >>>>> >> >> >>>>> -neutronsharc >> >> >>>> >> >> >>
