I have a node that I restarted and it's not joining the ensemble. I ask it 'stat' and it says it isn't serving any requests. Here's a snippet from the log:
2011-10-21 22:46:23,166 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 800 2011-10-21 22:46:23,167 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:46:23,169 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:46:23,970 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 1600 2011-10-21 22:46:23,971 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:46:23,973 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:46:25,290 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:44897 2011-10-21 22:46:25,574 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 3200 2011-10-21 22:46:25,575 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:46:25,577 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:46:28,000 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing ruok command from /127.0.0.1:44897 2011-10-21 22:46:28,001 - INFO [Thread-6:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:44897 (no session established for client) 2011-10-21 22:46:28,778 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 6400 2011-10-21 22:46:28,779 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:46:28,781 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:46:29,929 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:44898 2011-10-21 22:46:30,871 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing stat command from /127.0.0.1:44898 2011-10-21 22:46:30,873 - INFO [Thread-7:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:44898 (no session established for client) 2011-10-21 22:46:35,182 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 12800 2011-10-21 22:46:35,183 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:46:35,185 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:46:47,987 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 25600 2011-10-21 22:46:47,987 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:46:47,990 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:47:13,593 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 51200 2011-10-21 22:47:13,594 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:47:13,596 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:48:04,803 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 60000 2011-10-21 22:48:04,804 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:48:04,806 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:48:24,886 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:32954 2011-10-21 22:48:26,380 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing stat command from /127.0.0.1:32954 2011-10-21 22:48:26,381 - INFO [Thread-8:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:32954 (no session established for client) 2011-10-21 22:49:04,815 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 60000 2011-10-21 22:49:04,815 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:49:04,818 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:50:04,827 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 60000 2011-10-21 22:50:04,827 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:50:04,829 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:51:04,838 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 60000 2011-10-21 22:51:04,839 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:51:04,841 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:51:44,890 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:32964 2011-10-21 22:51:46,207 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing stat command from /127.0.0.1:32964 2011-10-21 22:51:46,208 - INFO [Thread-9:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:32964 (no session established for client) 2011-10-21 22:52:04,851 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 60000 2011-10-21 22:52:04,852 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:52:04,854 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:53:04,863 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 60000 2011-10-21 22:53:04,863 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:53:04,866 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state) 2011-10-21 22:54:04,876 - INFO [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@697] - Notification time out: 60000 2011-10-21 22:54:04,876 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 12885265585 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state) 2011-10-21 22:54:04,879 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 8589935532 (n.zxid), 3 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
