Hi Flavio, The bug / patch is only for the C client, does this issue affect the java client as well?
On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <[email protected]> wrote: > I will definitely try and I'll post an update, thanks again Flavio. > > On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <[email protected]> wrote: > >> I think you're hitting this: >> >> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 < >> https://issues.apache.org/jira/browse/ZOOKEEPER-2466> >> >> which is resolved for 3.5.3 and trunk. You can try out the patch. >> >> -Flavio >> >> > On 29 Jul 2016, at 00:56, Benjamin Jaton <[email protected]> >> wrote: >> > >> > OK so I did another fresh install this time using 3.5.2-alpha, and the >> read >> > only client doesn't seem to try all the hosts available. >> > >> > I create my client with: >> > ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181, >> > 10.11.12.210:2181", 45000, this, true); >> > >> > In the following output, it only checks for 10.11.12.4:2181 and >> > 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where >> the >> > running server is. >> > >> > 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:zookeeper.version=3.5.2-alpha-1753710, built on 07/21/2016 >> > 16:24 GMT >> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >> host.name >> > =w-rli09-ben >> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:java.version=1.8.0_11 >> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:java.vendor=Oracle Corporation >> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:java.home=/usr/local/apps/jdk1.8.0_11/jre >> > 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > >> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > >> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:java.io.tmpdir=/tmp >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:java.compiler=<NA> >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >> os.name >> > =Linux >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:os.arch=amd64 >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:os.version=3.8.0-44-generic >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >> user.name >> > =benji >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:user.home=/home/benji >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:user.dir=/home/benji/workspace-trunk/testzkclient >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:os.memory.free=376MB >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:os.memory.max=5358MB >> > 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> > environment:os.memory.total=378MB >> > *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client >> > connection, connectString=10.11.9.70:2181 >> > <http://10.11.9.70:2181>,10.11.12.4:2181 >> > <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181> >> > sessionTimeout=45000 watcher=testzkclient.TestZkReadOnlyClient@5010be6* >> > 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer >> is >> > 4194304 >> > Successfully established the connection with ZooKeeper >> > *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO >> > org.apache.zookeeper.ClientCnxn - Opening socket connection to server >> > 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will >> not >> > attempt to authenticate using SASL (unknown error)* >> > 43 [main-SendThread(10.11.12.4:2181)] WARN >> org.apache.zookeeper.ClientCnxn >> > - Session 0x0 for server null, unexpected error, closing socket >> connection >> > and attempting reconnect >> > java.net.ConnectException: Connection refused >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> > at >> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >> > at >> > >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) >> > at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >> > 45 [main-SendThread(10.11.12.4:2181)] DEBUG >> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during >> > shutdown input >> > java.nio.channels.ClosedChannelException >> > at >> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >> > at >> > >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) >> > at >> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >> > at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >> > 45 [main-SendThread(10.11.12.4:2181)] DEBUG >> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during >> > shutdown output >> > java.nio.channels.ClosedChannelException >> > at >> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >> > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) >> > at >> > >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) >> > at >> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >> > at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >> > *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)] >> INFO >> > org.apache.zookeeper.ClientCnxn - Opening socket connection to server >> > 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>. >> > Will not attempt to authenticate using SASL (unknown error)* >> > 150 [main-SendThread(10.11.12.210:2181)] WARN >> > org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected >> > error, closing socket connection and attempting reconnect >> > java.net.ConnectException: Connection refused >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> > at >> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >> > at >> > >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) >> > at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >> > 150 [main-SendThread(10.11.12.210:2181)] DEBUG >> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during >> > shutdown input >> > java.nio.channels.ClosedChannelException >> > at >> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >> > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >> > at >> > >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) >> > at >> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >> > at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >> > 150 [main-SendThread(10.11.12.210:2181)] DEBUG >> > org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception during >> > shutdown output >> > java.nio.channels.ClosedChannelException >> > at >> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >> > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) >> > at >> > >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) >> > at >> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >> > at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >> > *Exception in thread "main" >> > org.apache.zookeeper.KeeperException$ConnectionLossException: >> > KeeperErrorCode = ConnectionLoss for /* >> > at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> > at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956) >> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985) >> > at >> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26) >> > at >> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13) >> > >> > >> > Does anybody else have issues with the readonly mode? >> > >> > Thanks >> > Benjamin >> > >> > >> > On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton < >> [email protected]> >> > wrote: >> > >> >> Very interesting, thank you Flavio. >> >> >> >> That should not happen because I have actually never started (even >> never >> >> created) the other servers. This server is the first one the client >> could >> >> have possibly ever seen. Maybe the server has to have been in a >> functioning >> >> ensemble at least once before being able to be read only able? >> >> I will recheck all my setup and do it again. >> >> Thanks again, >> >> Ben >> >> >> >> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <[email protected]> >> wrote: >> >> >> >>> From the logs, it looks like the server the client is trying to >> connect >> >>> to is behind compared to last zxid the client has seen: >> >>> >> >>> Refusing session request for client /10.11.12.4:49237 as it >> has >> >>> seen zxid 0x3 our last zxid is 0x0 client must try another server >> >>> >> >>> I believe the behavior we currently have is that a client can connect >> to >> >>> an RO server, but the server must have a view at least as recent as >> the >> >>> server. >> >>> >> >>> -Flavio >> >>> >> >>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <[email protected]> >> >>> wrote: >> >>>> >> >>>> Hello, >> >>>> >> >>>> I'm having problems using ZK in read-only mode. >> >>>> I am using a fresh install of ZK 3.5.1-alpha where I configure the >> >>> installation to be in a 3 nodes ensemble. >> >>>> I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true. >> >>>> >> >>>> I see that the node QA-E8WIN11 starts the RO server properly: >> >>>> 2016-07-12 16:10:10,955 [myid:1] - INFO >> >>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started >> >>>> >> >>>> Then I make a read-only enabled client: >> >>>> >> >>>> ZooKeeper zk = new >> >>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000, >> this, >> >>> true); >> >>>> System.out.println("Successfully established the connection >> >>> with ZooKeeper"); >> >>>> >> >>>> zk.getData("/", false, null); >> >>>> System.out.println("Done."); >> >>>> >> >>>> Note that only QA-E8WIN11 is started, the other 2 are not. >> >>>> >> >>>> When the ZK client picks QA-E8WIN11 first, it works. >> >>>> But when it picks a different node, it fails with: >> >>>> >> >>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client >> >>> connection, >> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181 >> >>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145 >> >>>> Successfully established the connection with ZooKeeper >> >>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket >> >>> connection to server QA-E8WIN13/10.11.8.230:2181 < >> >>> http://10.11.8.230:2181/>. Will not attempt to authenticate using >> SASL >> >>> (unknown error) >> >>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session >> >>> timed out, have not heard from server in 15001ms for sessionid 0x0, >> closing >> >>> socket connection and attempting reconnect >> >>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket >> >>> connection to server QA-E8WIN12/10.11.8.232:2181 < >> >>> http://10.11.8.232:2181/>. Will not attempt to authenticate using >> SASL >> >>> (unknown error) >> >>>> Exception in thread "main" >> >>> org.apache.zookeeper.KeeperException$ConnectionLossException: >> >>> KeeperErrorCode = ConnectionLoss for / >> >>>> at >> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> >>>> at >> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753) >> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782) >> >>>> at TestZkReadOnly.run(TestZkReadOnly.java:26) >> >>>> at TestZkReadOnly.main(TestZkReadOnly.java:16) >> >>>> >> >>>> I'm attaching all the logs. The server and the client are on the same >> >>> machine QA-E8WIN11. >> >>>> >> >>>> Any help would be greatly appreciated! >> >>>> >> >>>> Thanks, >> >>>> Benjamin Jaton >> >>>> <zkclient.log><zkserver.log> >> >>> >> >>> >> >> >> >> >
