Bozhen Liu created ZOOKEEPER-3819:
-------------------------------------
Summary: Potential Races on DataTree
Key: ZOOKEEPER-3819
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3819
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.6.1, 3.5.4
Reporter: Bozhen Liu
We run our static race detector on ZooKeeper and discovered three methods in
class org/apache/zookeeper/server/DataTree may have concurrent accesses on
ephemerals without proper lock protection. The three methods are createNode(),
killSession() and deserialize(). They all read and write the ephemerals in the
similar way without common lock from get() to put():
{code:java}
HashSet<String> list = ephemerals.get(eowner);
if (list == null){
list = new HashSet<String>();
ephemerals.put(eowner, list);
}
list.add(path);
{code}
This is similar to ZOOKEEPER-3102.
The possible traces that lead to the race are (v3.5.4, probably the same in
v3.6.1):
=> Race: org/apache/zookeeper/server/DataTree.java/util/Map
(org/apache/zookeeper/server/DataTree:478 (TID: 2190) ,
org/apache/zookeeper/server/DataTree:1215 (TID: 9592) )
Trace 1st node: (TID: 2190)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun
from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from
org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig
(line 200)
-> Call java/lang/Thread.start from
org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from
java/lang/Thread.start (line -1)
*> Thread (2190) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeer>.run (line 1167)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.run from
java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.pRequest from
org/apache/zookeeper/server/PrepRequestProcessor.run (line 145)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from
org/apache/zookeeper/server/PrepRequestProcessor.pRequest (line 906)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from
org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.createNode from
org/apache/zookeeper/server/DataTree.processTxn (line 786)
=> Read on org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in
org/apache/zookeeper/server/DataTree.createNode (line 478)
Trace 2st node: (TID: 9592)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun
from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from
org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig
(line 200)
-> Call java/lang/Thread.start from
org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from
java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/Leader.lead from
org/apache/zookeeper/server/quorum/QuorumPeer.run (line 1227)
*> Thread (6402) created by
Application,Lorg/apache/zookeeper/server/quorum/Leader>.lead (line 448)
-> Call org/apache/zookeeper/server/quorum/Leader$LearnerCnxAcceptor.run from
java/lang/Thread.start (line -1)
*> Thread (7676) created by
Application,Lorg/apache/zookeeper/server/quorum/Leader$LearnerCnxAcceptor>.run
(line 386)
-> Call org/apache/zookeeper/server/quorum/LearnerHandler.run from
java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/Leader.processAck from
org/apache/zookeeper/server/quorum/LearnerHandler.run (line 559)
-> Call org/apache/zookeeper/server/quorum/Leader.tryToCommit from
org/apache/zookeeper/server/quorum/Leader.processAck (line 863)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.processReconfig from
org/apache/zookeeper/server/quorum/Leader.tryToCommit (line 779)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.restartLeaderElection
from org/apache/zookeeper/server/quorum/QuorumPeer.processReconfig (line 1871)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.startLeaderElection from
org/apache/zookeeper/server/quorum/QuorumPeer.restartLeaderElection (line 1530)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.getLastLoggedZxid from
org/apache/zookeeper/server/quorum/QuorumPeer.startLeaderElection (line 917)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.loadDataBase from
org/apache/zookeeper/server/quorum/QuorumPeer.getLastLoggedZxid (line 1009)
-> Call org/apache/zookeeper/server/ZKDatabase.loadDataBase from
org/apache/zookeeper/server/quorum/QuorumPeer.loadDataBase (line 867)
-> Call org/apache/zookeeper/server/persistence/FileTxnSnapLog.restore from
org/apache/zookeeper/server/ZKDatabase.loadDataBase (line 240)
-> Call org/apache/zookeeper/server/persistence/FileSnap.deserialize from
org/apache/zookeeper/server/persistence/FileTxnSnapLog.restore (line 200)
-> Call org/apache/zookeeper/server/persistence/FileSnap.deserialize from
org/apache/zookeeper/server/persistence/FileSnap.deserialize (line 87)
-> Call org/apache/zookeeper/server/util/SerializeUtils.deserializeSnapshot
from org/apache/zookeeper/server/persistence/FileSnap.deserialize (line 122)
-> Call org/apache/zookeeper/server/DataTree.deserialize from
org/apache/zookeeper/server/util/SerializeUtils.deserializeSnapshot (line 141)
=> Write to org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in
org/apache/zookeeper/server/DataTree.deserialize (line 1215)
--------------------------------------------------------------------------------------------------------------------------------
=> Race: org/apache/zookeeper/server/DataTree.java/util/Map
(org/apache/zookeeper/server/DataTree:478 (TID: 2190) ,
org/apache/zookeeper/server/DataTree:1005 (TID: 2835) )
Trace 1st node: (TID: 2190)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun
from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from
org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig
(line 200)
-> Call java/lang/Thread.start from
org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from
java/lang/Thread.start (line -1)
*> Thread (2190) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeer>.run (line 1167)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.run from
java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.pRequest from
org/apache/zookeeper/server/PrepRequestProcessor.run (line 145)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from
org/apache/zookeeper/server/PrepRequestProcessor.pRequest (line 906)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from
org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.createNode from
org/apache/zookeeper/server/DataTree.processTxn (line 786)
=> Read on org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in
org/apache/zookeeper/server/DataTree.createNode (line 478)
Trace 2st node: (TID: 2835)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun
from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from
org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig
(line 200)
-> Call java/lang/Thread.start from
org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from
java/lang/Thread.start (line -1)
*> Thread (2190) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeer>.run (line 1167)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer$1.run from
java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/ReadOnlyZooKeeperServer.startup
from org/apache/zookeeper/server/quorum/QuorumPeer$1.run (line 1157)
-> Call org/apache/zookeeper/server/ZooKeeperServer.startup from
org/apache/zookeeper/server/quorum/ReadOnlyZooKeeperServer.startup (line 70)
-> Call
org/apache/zookeeper/server/quorum/ReadOnlyZooKeeperServer.setupRequestProcessors
from org/apache/zookeeper/server/ZooKeeperServer.startup (line 453)
*> Thread (2835) created by
Application,Lorg/apache/zookeeper/server/quorum/ReadOnlyZooKeeperServer>.setupRequestProcessors
(line 59)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.run from
java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.pRequest from
org/apache/zookeeper/server/PrepRequestProcessor.run (line 145)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from
org/apache/zookeeper/server/PrepRequestProcessor.pRequest (line 906)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from
org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.killSession from
org/apache/zookeeper/server/DataTree.processTxn (line 854)
=> Write to org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in
org/apache/zookeeper/server/DataTree.killSession (line 1005)
--------------------------------------------------------------------------------------------------------------------------------
=> Race: org/apache/zookeeper/server/DataTree.java/util/Map
(org/apache/zookeeper/server/DataTree:478 (TID: 2190) ,
org/apache/zookeeper/server/DataTree:481 (TID: 13154) )
Trace 1st node: (TID: 2190)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun
from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from
org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig
(line 200)
-> Call java/lang/Thread.start from
org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from
java/lang/Thread.start (line -1)
*> Thread (2190) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeer>.run (line 1167)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.run from
java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.pRequest from
org/apache/zookeeper/server/PrepRequestProcessor.run (line 145)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from
org/apache/zookeeper/server/PrepRequestProcessor.pRequest (line 906)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from
org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.createNode from
org/apache/zookeeper/server/DataTree.processTxn (line 786)
=> Read on org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in
org/apache/zookeeper/server/DataTree.createNode (line 478)
Trace 2st node: (TID: 13154)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun
from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from
org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by
Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig
(line 200)
-> Call java/lang/Thread.start from
org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from
java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/Observer.observeLeader from
org/apache/zookeeper/server/quorum/QuorumPeer.run (line 1201)
-> Call org/apache/zookeeper/server/quorum/Learner.syncWithLeader from
org/apache/zookeeper/server/quorum/Observer.observeLeader (line 74)
-> Call org/apache/zookeeper/server/ZooKeeperServer.startup from
org/apache/zookeeper/server/quorum/Learner.syncWithLeader (line 559)
-> Call
org/apache/zookeeper/server/quorum/ObserverZooKeeperServer.setupRequestProcessors
from org/apache/zookeeper/server/ZooKeeperServer.startup (line 453)
*> Thread (8653) created by
Application,Lorg/apache/zookeeper/server/quorum/ObserverZooKeeperServer>.setupRequestProcessors
(line 97)
-> Call java/lang/Thread.start from
org/apache/zookeeper/server/quorum/CommitProcessor.start (line 263)
-> Call org/apache/zookeeper/server/quorum/CommitProcessor.run from
java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/CommitProcessor.sendToNextProcessor
from org/apache/zookeeper/server/quorum/CommitProcessor.run (line 180)
-> Call org/apache/zookeeper/server/WorkerService.schedule from
org/apache/zookeeper/server/quorum/CommitProcessor.sendToNextProcessor (line
272)
*> Thread (13154) created by
Application,Lorg/apache/zookeeper/server/WorkerService>.schedule (line 128)
-> Call org/apache/zookeeper/server/WorkerService$ScheduledWorkRequest.run
from java/lang/Thread.start (line -1)
-> Call
org/apache/zookeeper/server/quorum/CommitProcessor$CommitWorkRequest.doWork
from org/apache/zookeeper/server/WorkerService$ScheduledWorkRequest.run (line
162)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from
org/apache/zookeeper/server/quorum/CommitProcessor$CommitWorkRequest.doWork
(line 297)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from
org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from
org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.createNode from
org/apache/zookeeper/server/DataTree.processTxn (line 786)
=> Write to org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in
org/apache/zookeeper/server/DataTree.createNode (line 481)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
