Re: Spew after call to close
No worries, let us know if something else pops up. Patrick On Tue, Sep 7, 2010 at 3:10 PM, Stack st...@duboce.net wrote: Nevermind. I figured it. It was an hbase issue. We were leaking a client reference. Sorry for the noise, St.Ack On Sat, Sep 4, 2010 at 10:58 AM, Stack st...@duboce.net wrote: Thats right -- client is shutdown first, then server... How do I stop the client trying to come back from the dead? Good on you Mahadev? St.Ack On Fri, Sep 3, 2010 at 8:36 PM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Stack, Looks like you are shutting down the server and shutting down the client at the same time? Is that the issue? Thanks mahadev On 9/3/10 4:47 PM, Stack st...@duboce.net wrote: Have you fellas seen this before? I call close on zookeeper but it insists on doing the below exceptions. Why is it doing this 'Session 0x12ad9dccda30002 for server null, unexpected error, closing socket connection and attempting reconnect'? This would seem to come after the close has been noticed and looking in code, i'd think we'd not do this since the close flag should be set to true post call to close? Thanks lads (The below looks ugly in our logs... this is zk 3.3.1), St.Ack 2010-09-03 16:09:52,369 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid 0x12ad9dccda30001 2010-09-03 16:09:52,369 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ad9dccda30001, likely server has closed socket, closing socket connection and attempting reconnect 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ad9dccda30002, likely server has closed socket, closing socket connection and attempting reconnect 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory exited run method 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor exited loop! 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor exited! 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request processor complete 2010-09-03 16:09:52,470 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2010-09-03 16:09:52,470 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received Disconnected from ZooKeeper, ignoring 2010-09-03 16:09:52,471 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2010-09-03 16:09:52,471 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received Disconnected from ZooKeeper, ignoring 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30001 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30002 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30001 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-03 16:09:54,000 INFO
Re: Spew after call to close
No worries, let us know if something else pops up. Patrick On Tue, Sep 7, 2010 at 3:10 PM, Stack st...@duboce.net wrote: Nevermind. I figured it. It was an hbase issue. We were leaking a client reference. Sorry for the noise, St.Ack On Sat, Sep 4, 2010 at 10:58 AM, Stack st...@duboce.net wrote: Thats right -- client is shutdown first, then server... How do I stop the client trying to come back from the dead? Good on you Mahadev? St.Ack On Fri, Sep 3, 2010 at 8:36 PM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Stack, Looks like you are shutting down the server and shutting down the client at the same time? Is that the issue? Thanks mahadev On 9/3/10 4:47 PM, Stack st...@duboce.net wrote: Have you fellas seen this before? I call close on zookeeper but it insists on doing the below exceptions. Why is it doing this 'Session 0x12ad9dccda30002 for server null, unexpected error, closing socket connection and attempting reconnect'? This would seem to come after the close has been noticed and looking in code, i'd think we'd not do this since the close flag should be set to true post call to close? Thanks lads (The below looks ugly in our logs... this is zk 3.3.1), St.Ack 2010-09-03 16:09:52,369 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid 0x12ad9dccda30001 2010-09-03 16:09:52,369 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ad9dccda30001, likely server has closed socket, closing socket connection and attempting reconnect 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ad9dccda30002, likely server has closed socket, closing socket connection and attempting reconnect 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory exited run method 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor exited loop! 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor exited! 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request processor complete 2010-09-03 16:09:52,470 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2010-09-03 16:09:52,470 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received Disconnected from ZooKeeper, ignoring 2010-09-03 16:09:52,471 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2010-09-03 16:09:52,471 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received Disconnected from ZooKeeper, ignoring 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30001 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30002 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30001 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-03 16:09:54,000 INFO
Re: Spew after call to close
Hi Stack, Looks like you are shutting down the server and shutting down the client at the same time? Is that the issue? Thanks mahadev On 9/3/10 4:47 PM, Stack st...@duboce.net wrote: Have you fellas seen this before? I call close on zookeeper but it insists on doing the below exceptions. Why is it doing this 'Session 0x12ad9dccda30002 for server null, unexpected error, closing socket connection and attempting reconnect'? This would seem to come after the close has been noticed and looking in code, i'd think we'd not do this since the close flag should be set to true post call to close? Thanks lads (The below looks ugly in our logs... this is zk 3.3.1), St.Ack 2010-09-03 16:09:52,369 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid 0x12ad9dccda30001 2010-09-03 16:09:52,369 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ad9dccda30001, likely server has closed socket, closing socket connection and attempting reconnect 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ad9dccda30002, likely server has closed socket, closing socket connection and attempting reconnect 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory exited run method 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor exited loop! 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor exited! 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request processor complete 2010-09-03 16:09:52,470 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2010-09-03 16:09:52,470 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received Disconnected from ZooKeeper, ignoring 2010-09-03 16:09:52,471 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2010-09-03 16:09:52,471 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: localhost:/hbase Received Disconnected from ZooKeeper, ignoring 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30001 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30002 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30001 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) 2010-09-03 16:09:54,000 INFO org.apache.zookeeper.server.SessionTrackerImpl: SessionTrackerImpl exited loop! 2010-09-03 16:09:54,002 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Closed zookeeper sessionid=0x12ad9dccda30001 2010-09-03 16:09:54,129 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181 2010-09-03 16:09:54,130 WARN org.apache.zookeeper.ClientCnxn: Session 0x12ad9dccda30002 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at
