Thanks for looking at this Camille and Benjamin, setup: There are 5 machines, 2 hosting clients and 3 hosting servers. There is one client process on each of the client machines The client process has 20 threads, each thread with 500 sessions. So I have a total of 20K clients, so it isn't that high really
Hardware Two proc Intel® Xeon® Processor L5420 (total 8 cores) 8G RAM The workload is fairly simple: All sessions do is keep a watch on a node. Once the watch fires, the client reads the contents of the node and puts the watch again. There is one thread that is periodically updating the node being watched (once every 30s - so very infrequent) When the system starts off, things are fine, then a few timers starts missing and eventually there are lots of expired connections. The logs are really long, but pretty much repetitive, so I am attaching the tail of the logs. The client timeout is 300s JVM Parameters -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:MaxGCPauseMillis=50 -Dzookeeper.globalOutstandingLimit=30000 -Xms6000m -Xmx6000m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8180 I have GC logging turned on. I am not seeing long GC pauses, so I don't think that's it. Next steps I am trying 1. Look at the CPU utilization on the server machines 2. If the CPU is pegged at 100%, add some additional tracing in the server to validate my hypothesis that the session tracker is getting overwhelmed If you folks have any other suggestions, that would greatly help. I started working with zookeeper a couple of weeks ago so it is very likely I might be missing something obvious. Thanks! Vishal -----Original Message----- From: Benjamin Reed [mailto:br...@apache.org] Sent: Sunday, May 29, 2011 8:42 PM To: dev@zookeeper.apache.org Subject: Re: Discussion on supporting a large number of clients for a zk ensemble i second camille's suggestion. i also know there are other people looking into using zookeeper with a large number of clients, so it would be good to figure out what are the limits and then how to cross them. i like your proposed solutions, but i would rather start down that road after we have resolved the issues that we can for the normal clients. ben On Fri, May 27, 2011 at 4:23 PM, Fournier, Camille F. [Tech] <camille.fourn...@gs.com> wrote: > I would recommend that you spend some time making sure that your guess about > the cause is correct before trying to design solutions to the problem. Can > you provide us some hard numbers, logs, and configuration information? It's > always possible that some aspect of your configuration that you hadn't > considered important is in fact the trigger here. > > Thanks, > Camille > > -----Original Message----- > From: Vishal Kathuria [mailto:vishal.kathu...@fb.com] > Sent: Friday, May 27, 2011 6:32 PM > To: dev@zookeeper.apache.org > Subject: Discussion on supporting a large number of clients for a zk > ensemble > > Hi Folks, > I wanted to start a discussion on how we can support a large number of > clients in zookeeper. I am at facebook and we are using zookeeper for > quite a few projects. There are a couple of projects where we are > designing for a large number of clients. The projects are > > > 1. Building a directory service for holding configuration information > (lookup table for which node to go to for a given key). > > 2. For HDFS clients, where clients lookup zookeeper for the > current namenode > > This information changes infrequently and is small, so update rate or size of > data is not an issue. > > The key challenge is to support that large a number of clients (30K to start > with, but eventually could be 100K). A big chunk of the clients can try to > connect/disconnect at the same time - so herd effect can happen. > > I was trying out a 3 node ensemble. I noticed that with about 20K clients, > there we quite a few session expires and disconnects. > I looked through the code briefly and since all the pings are eventually > handled by the leader, my guess is that the leader thread is not keeping up. > I haven't yet do the instrumentation/tracing to validate this. > > I have been thinking about how to improve this and thought of the following > solution. I am trying to hit 2 goals with this. > > 1. Make it possible to have a very large number of clients (each client > has a watch) without losing connections too often. > > 2. Improve how quickly a large number of clients can connect. > > Solution > > 1. The idea is to introduce a new type of session - "local" session. A > "local" session doesn't have a full functionality of a normal session. > > 2. Local sessions cannot create ephemeral nodes. > > 3. Once a local session is lost, you cannot re-establish it using the > session-id/password. The session and its watches are gone for good. > > 4. When a local session connects, the session info is only maintained > on the zookeeper server that it is connected to. The leader is not aware of > the creation of such a session and there is no state written to disk. > > 5. The pings and expiration is handled by the server that the session > is connected to. > > With the above changes, it should be easy to scale ZK by adding more > learners, which manage the "local" sessions independently. Also, the rate at > which you can establish "local" sessions, would be significantly higher than > the normal sessions. > > Would like to stir up a discussion on whether this is the best way to achieve > these goals or if I am missing simpler ways of accomplishing this. > > Thanks! > Vishal > > . > >
2011-05-30 09:39:19,846:25469(0x74a17940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:19,941:25469(0x467d7940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:20,111:25469(0x5235d940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 102ms 2011-05-30 09:42:11,703:25469(0x74a17940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 71952ms 2011-05-30 09:42:11,703:25469(0x74a17940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.58.79:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 71952ms) 2011-05-30 09:39:20,604:25469(0x66bb7940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:20,650:25469(0x8d3fe940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:42:11,719:25469(0xa350d940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 73349ms 2011-05-30 09:42:11,721:25469(0x8d3fe940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 71165ms 2011-05-30 09:39:20,670:25469(0x9dbfd940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 104ms 2011-05-30 09:39:17,847:25469(0x273f5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:20,675:25469(0x974f5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:39:20,759:25469(0x4e7e7940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:42:11,782:25469(0x974f5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 71342ms 2011-05-30 09:39:20,981:25469(0x934ed940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:39:21,061:25469(0xeb4e0940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:42:11,851:25469(0x934ed940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 71105ms 2011-05-30 09:39:21,390:25469(0x74f9e940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:42:11,685:25469(0xa6513940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 75611ms) 2011-05-30 09:42:11,312:25469(0x54c70940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 76225ms) 2011-05-30 09:42:11,858:25469(0xeb4e0940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 70969ms 2011-05-30 09:42:11,858:25469(0xeb4e0940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 70969ms) 2011-05-30 09:42:11,874:25469(0x273f5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 74123ms 2011-05-30 09:42:11,874:25469(0x273f5940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.58.79:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 74124ms) 2011-05-30 09:42:11,888:25469(0x54c70940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 76923ms 2011-05-30 09:42:11,888:25469(0x273f5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 74137ms 2011-05-30 09:42:11,907:25469(0xa6513940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 76236ms 2011-05-30 09:42:11,309:25469(0xba53b940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 74220ms 2011-05-30 09:42:11,907:25469(0xa93ff940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 75585ms 2011-05-30 09:42:11,907:25469(0xba53b940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 74221ms) 2011-05-30 09:42:11,907:25469(0xa93ff940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.56.65:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 75586ms) 2011-05-30 09:42:11,907:25469(0xa93ff940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 75585ms 2011-05-30 09:42:11,918:25469(0x217fd940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 73775ms 2011-05-30 09:39:21,473:25469(0x9f505940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:39:21,556:25469(0x914e9940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:42:11,929:25469(0xeb4e0940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 71038ms 2011-05-30 09:39:22,040:25469(0x8fce6940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:22,431:25469(0x8d4e1940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:22,868:25469(0x8bcde940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:42:11,988:25469(0x8bcde940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 69260ms 2011-05-30 09:39:23,091:25469(0x874d5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:39:23,102:25469(0x8a4db940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 110ms 2011-05-30 09:42:11,991:25469(0x874d5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 69003ms 2011-05-30 09:42:11,999:25469(0x8a4db940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 69004ms 2011-05-30 09:39:23,371:25469(0x884d7940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:23,943:25469(0x7f3da940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 103ms 2011-05-30 09:42:12,037:25469(0x974f5940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 71342ms) 2011-05-30 09:39:23,947:25469(0x7b3e2940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:42:12,073:25469(0x7f3da940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 68220ms 2011-05-30 09:42:12,083:25469(0x7f3da940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.56.65:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 68221ms) 2011-05-30 09:42:12,086:25469(0x884d7940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 68814ms 2011-05-30 09:42:12,091:25469(0x7f3da940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 68238ms 2011-05-30 09:42:11,988:25469(0x8fce6940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 70132ms 2011-05-30 09:39:24,216:25469(0x3e7df940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 103ms 2011-05-30 09:42:11,988:25469(0x914e9940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 70551ms 2011-05-30 09:39:24,292:25469(0x9bcfe940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:25,145:25469(0x854d1940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:42:12,091:25469(0x974f5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 71652ms 2011-05-30 09:42:12,091:25469(0x9bcfe940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 67919ms 2011-05-30 09:39:25,320:25469(0x84cd0940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:25,172:25469(0x83cce940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 100ms 2011-05-30 09:39:26,251:25469(0x814c9940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 101ms 2011-05-30 09:39:26,090:25469(0x904e7940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 103ms 2011-05-30 09:42:12,091:25469(0xa350d940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.58.79:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 73349ms) 2011-05-30 09:42:12,125:25469(0x83cce940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 67067ms
2011-05-29 15:52:26,266:3417(0x21fd9940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.56.65:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 152ms) 2011-05-29 15:52:26,266:3417(0x21fd9940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 61984ms 2011-05-29 15:51:24,625:3417(0x287d3940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 110ms 2011-05-29 15:51:24,650:3417(0x3bf6f940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 108ms 2011-05-29 15:51:24,605:3417(0xe46a9940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 98ms 2011-05-29 15:52:26,269:3417(0x287d3940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.56.65:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 111ms) 2011-05-29 15:51:24,658:3417(0x2a7d7940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 110ms 2011-05-29 15:51:24,658:3417(0x37f67940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 2ms) 2011-05-29 15:51:24,664:3417(0x197b5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 108ms 2011-05-29 15:51:24,675:3417(0xf9775940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 105ms 2011-05-29 15:51:24,450:3417(0xf73d5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 141ms 2011-05-29 15:51:24,458:3417(0x5179a940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 267ms 2011-05-29 15:51:24,683:3417(0x321e7940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 174ms 2011-05-29 15:52:26,269:3417(0x37f67940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 61613ms 2011-05-29 15:49:45,214:3417(0x117a5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 152ms 2011-05-29 15:51:24,844:3417(0x24fcc940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 259ms 2011-05-29 15:49:45,492:3417(0x1e7f1940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 181ms 2011-05-29 15:52:26,269:3417(0x117a5940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 61206ms 2011-05-29 15:52:26,269:3417(0x1e7f1940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 60958ms 2011-05-29 15:51:24,691:3417(0x36f65940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 118ms 2011-05-29 15:51:24,844:3417(0x41f7b940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 399ms 2011-05-29 15:49:45,594:3417(0xf7fdc940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 121ms 2011-05-29 15:52:26,269:3417(0x1e7f1940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.58.79:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 60959ms) 2011-05-29 15:51:24,892:3417(0x19ff6940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 127ms 2011-05-29 15:52:26,269:3417(0x30f59940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 61020ms 2011-05-29 15:52:26,269:3417(0x36f65940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.56.65:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 118ms) 2011-05-29 15:51:24,850:3417(0x47f87940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 103ms 2011-05-29 15:51:24,786:3417(0xf4bd0940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 135ms 2011-05-29 15:52:26,269:3417(0x1e7f1940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 60958ms 2011-05-29 15:51:24,892:3417(0x217c5940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.58.79:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 146ms) 2011-05-29 15:51:24,844:3417(0xe47ff940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 123ms 2011-05-29 15:52:26,269:3417(0xf4bd0940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.58.79:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 135ms) 2011-05-29 15:51:24,969:3417(0x3976a940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 410ms 2011-05-29 15:51:24,844:3417(0xf5bd2940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 596ms 2011-05-29 15:52:26,269:3417(0x30f59940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.58.79:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 61021ms) 2011-05-29 15:51:24,981:3417(0x247cb940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 115ms 2011-05-29 15:51:24,997:3417(0x17ef940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 104ms 2011-05-29 15:51:25,002:3417(0x37f3940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 108ms 2011-05-29 15:49:45,637:3417(0xfa777940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 114ms 2011-05-29 15:52:26,269:3417(0x17ef940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 104ms) 2011-05-29 15:51:25,091:3417(0x157ad940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 126ms 2011-05-29 15:51:25,099:3417(0x127a7940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.58.79:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 107ms) 2011-05-29 15:51:25,193:3417(0x2bf4f940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.56.65:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 4ms) 2011-05-29 15:51:25,198:3417(0x22ffa940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 5ms) 2011-05-29 15:52:26,269:3417(0x157ad940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.56.65:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 125ms) 2011-05-29 15:51:25,219:3417(0x578d940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 247ms 2011-05-29 15:52:26,269:3417(0xf576d940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 60763ms 2011-05-29 15:49:46,134:3417(0xe1745940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 143ms 2011-05-29 15:52:26,275:3417(0xf576d940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.56.65:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 60763ms) 2011-05-29 15:52:26,275:3417(0xe1745940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 60284ms 2011-05-29 15:52:26,277:3417(0x2ff57940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 61009ms 2011-05-29 15:52:26,275:3417(0xe1745940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 60283ms) 2011-05-29 15:52:26,280:3417(0x157ad940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 61314ms 2011-05-29 15:52:26,285:3417(0x247fd940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 61005ms 2011-05-29 15:52:26,287:3417(0x10b01940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 99ms 2011-05-29 15:52:26,275:3417(0xf576d940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 60768ms 2011-05-29 15:48:55,451:3417(0x4a575940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 139ms 2011-05-29 15:52:26,295:3417(0xe3f4a940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 60260ms 2011-05-29 15:52:26,311:3417(0x4a575940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 110999ms 2011-05-29 15:52:26,311:3417(0x4a575940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.57.73:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 111000ms) 2011-05-29 15:52:26,318:3417(0xdffff940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 108312ms 2011-05-29 15:52:26,321:3417(0xe3f4a940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.58.79:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 60260ms) 2011-05-29 15:51:25,283:3417(0xb799940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 119ms 2011-05-29 15:52:26,324:3417(0x83bfe940):ZOO_WARN@zookeeper_interest@1084: Exceeded deadline by 116785ms 2011-05-29 15:52:26,339:3417(0xb799940):ZOO_ERROR@handle_socket_error_msg@1137: Socket [10.47.56.65:2181] zk retcode=-7, errno=110(Connection timed out): connection timed out (exceeded timeout by 120ms)
2011-05-30 09:47:00,055 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@389] - Processed session termination for sessionid: 0x2303d83faa6304f 2011-05-30 09:47:20,010 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437cd, timeout of 300000ms exceeded 2011-05-30 09:47:20,010 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437d8, timeout of 300000ms exceeded 2011-05-30 09:47:20,010 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231c5, timeout of 300000ms exceeded 2011-05-30 09:47:20,010 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa63083, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22b08, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22b04, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa6307b, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22b09, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22af9, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e4305d, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22b03, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa6307d, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa629cc, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa63087, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa6307f, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e4363b, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231d7, timeout of 300000ms exceeded 2011-05-30 09:47:20,011 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437d7, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231cd, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa62f19, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e43022, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437dd, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437df, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22b05, timeout of 300000ms exceeded 2011-05-30 09:47:20,003 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@389] - Processed session termination for sessionid: 0x3303d83abb231bb 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437cf, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e43062, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22b07, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231d1, timeout of 300000ms exceeded 2011-05-30 09:47:20,012 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22ae9, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa629d7, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa63078, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231bd, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231c7, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa6307c, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437dc, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437d5, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa62f1e, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa6306f, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231c9, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22afd, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437e3, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb22af1, timeout of 300000ms exceeded 2011-05-30 09:47:20,013 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231d0, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437ea, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231d9, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa63076, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437d1, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa63080, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e42fb2, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa63070, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437d9, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231cc, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa63085, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa629d8, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa6293a, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437d6, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437ec, timeout of 300000ms exceeded 2011-05-30 09:47:20,014 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437d3, timeout of 300000ms exceeded 2011-05-30 09:47:20,015 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231ce, timeout of 300000ms exceeded 2011-05-30 09:47:20,015 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa62f09, timeout of 300000ms exceeded 2011-05-30 09:47:20,015 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa629de, timeout of 300000ms exceeded 2011-05-30 09:47:20,015 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa629df, timeout of 300000ms exceeded 2011-05-30 09:47:20,015 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x3303d83abb231cf, timeout of 300000ms exceeded 2011-05-30 09:47:20,015 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x1303d83e3e437d0, timeout of 300000ms exceeded 2011-05-30 09:47:20,015 [myid:3] - INFO [SessionTracker:ZooKeeperServer@317] - Expiring session 0x2303d83faa63079, timeout of 300000ms exceeded
2011-05-30 09:29:16,879 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e41d3c for client /10.47.59.91:46892, probably expired 2011-05-30 09:29:16,880 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46881 which had sessionid 0x1303d83e3e4190a 2011-05-30 09:29:16,936 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e416ce at /10.47.59.91:46894 2011-05-30 09:29:16,937 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e416ce 2011-05-30 09:29:16,937 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46892 which had sessionid 0x1303d83e3e41d3c 2011-05-30 09:29:16,937 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e416ce for client /10.47.59.91:46894, probably expired 2011-05-30 09:29:16,983 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46894 which had sessionid 0x1303d83e3e416ce 2011-05-30 09:29:17,464 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46900 2011-05-30 09:29:17,536 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e426e4 at /10.47.59.91:46900 2011-05-30 09:29:17,537 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e426e4 2011-05-30 09:29:17,537 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e426e4 for client /10.47.59.91:46900, probably expired 2011-05-30 09:29:17,537 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46900 which had sessionid 0x1303d83e3e426e4 2011-05-30 09:29:17,584 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46905 2011-05-30 09:29:17,636 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46907 2011-05-30 09:29:17,665 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e41771 at /10.47.59.91:46907 2011-05-30 09:29:17,666 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e41771 2011-05-30 09:29:17,666 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e41771 for client /10.47.59.91:46907, probably expired 2011-05-30 09:29:17,666 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46907 which had sessionid 0x1303d83e3e41771 2011-05-30 09:29:17,719 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e42c58 at /10.47.59.91:46905 2011-05-30 09:29:17,719 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e42c58 2011-05-30 09:29:17,720 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e42c58 for client /10.47.59.91:46905, probably expired 2011-05-30 09:29:17,720 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46905 which had sessionid 0x1303d83e3e42c58 2011-05-30 09:29:17,960 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46917 2011-05-30 09:29:17,962 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e4268f at /10.47.59.91:46917 2011-05-30 09:29:17,962 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e4268f 2011-05-30 09:29:17,963 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e4268f for client /10.47.59.91:46917, probably expired 2011-05-30 09:29:17,963 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46917 which had sessionid 0x1303d83e3e4268f 2011-05-30 09:29:18,022 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46921 2011-05-30 09:29:18,165 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46925 2011-05-30 09:29:18,226 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e42685 at /10.47.59.91:46925 2011-05-30 09:29:18,227 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e42685 2011-05-30 09:29:18,227 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e42685 for client /10.47.59.91:46925, probably expired 2011-05-30 09:29:18,227 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46925 which had sessionid 0x1303d83e3e42685 2011-05-30 09:29:18,406 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46930 2011-05-30 09:29:18,418 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46931 2011-05-30 09:29:18,438 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e41b24 at /10.47.59.91:46930 2011-05-30 09:29:18,438 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e41b24 2011-05-30 09:29:18,438 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e41b24 for client /10.47.59.91:46930, probably expired 2011-05-30 09:29:18,439 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46930 which had sessionid 0x1303d83e3e41b24 2011-05-30 09:29:18,491 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46932 2011-05-30 09:29:18,500 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e41051 at /10.47.59.91:46931 2011-05-30 09:29:18,500 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e41051 2011-05-30 09:29:18,501 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e41051 for client /10.47.59.91:46931, probably expired 2011-05-30 09:29:18,501 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46931 which had sessionid 0x1303d83e3e41051 2011-05-30 09:29:18,561 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46933 2011-05-30 09:29:18,561 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46936 2011-05-30 09:29:18,561 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46937 2011-05-30 09:29:18,561 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46938 2011-05-30 09:29:18,565 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e42435 at /10.47.59.91:46936 2011-05-30 09:29:18,565 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e42435 2011-05-30 09:29:18,565 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e4271e at /10.47.59.91:46937 2011-05-30 09:29:18,566 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e4271e 2011-05-30 09:29:18,566 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e42435 for client /10.47.59.91:46936, probably expired 2011-05-30 09:29:18,566 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e4271e for client /10.47.59.91:46937, probably expired 2011-05-30 09:29:18,566 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46936 which had sessionid 0x1303d83e3e42435 2011-05-30 09:29:18,579 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46937 which had sessionid 0x1303d83e3e4271e 2011-05-30 09:29:18,584 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46940 2011-05-30 09:29:18,586 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.47.59.91:46941 2011-05-30 09:29:18,670 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e41a38 at /10.47.59.91:46941 2011-05-30 09:29:18,670 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e41a38 2011-05-30 09:29:18,671 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e41a38 for client /10.47.59.91:46941, probably expired 2011-05-30 09:29:18,671 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Closed socket connection for client /10.47.59.91:46941 which had sessionid 0x1303d83e3e41a38 2011-05-30 09:29:18,727 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e418e0 at /10.47.59.91:46940 2011-05-30 09:29:18,727 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e418e0 2011-05-30 09:29:18,727 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@794] - Client attempting to renew session 0x1303d83e3e418a1 at /10.47.59.91:46932 2011-05-30 09:29:18,727 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@103] - Revalidating client: 0x1303d83e3e418a1 2011-05-30 09:29:18,727 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@570] - Invalid session 0x1303d83e3e418e0 for client /10.47.59.91:46940, probably expired
