Update, I was confused by races of my own doing. Was this client previously connected when it failed the read only check? The server log is rejecting it because the client is ahead in zxid in the first set of logs. The other two servers are totally down and it's a new client when you see this issue?
On Jul 29, 2016 3:07 PM, "Camille Fournier" <[email protected]> wrote: > Ok yeah I think this is reproducible and a bug in the client connection > read-only logic. > > On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <[email protected]> > wrote: > >> I'm looking at the readonly mode code right now and it appears that the >> only way to set readonly mode is a global system property which means that >> the tests for this are only testing across 3 servers, all of which have >> readonly mode set. So, this MAY be a bug, but what a pain to figure out how >> to reproduce. >> >> Global system properties: not even once. >> >> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <[email protected]> >> wrote: >> >>> Not that I'm aware of... >>> >>> -Flavio >>> >>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <[email protected]> >>> wrote: >>> > >>> > Hi Flavio, >>> > >>> > The bug / patch is only for the C client, does this issue affect the >>> java >>> > client as well? >>> > >>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton < >>> [email protected]> >>> > wrote: >>> > >>> >> I will definitely try and I'll post an update, thanks again Flavio. >>> >> >>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <[email protected]> >>> wrote: >>> >> >>> >>> I think you're hitting this: >>> >>> >>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 < >>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466> >>> >>> >>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch. >>> >>> >>> >>> -Flavio >>> >>> >>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <[email protected]> >>> >>> wrote: >>> >>>> >>> >>>> OK so I did another fresh install this time using 3.5.2-alpha, and >>> the >>> >>> read >>> >>>> only client doesn't seem to try all the hosts available. >>> >>>> >>> >>>> I create my client with: >>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181, >>> >>>> 10.11.12.210:2181", 45000, this, true); >>> >>>> >>> >>>> In the following output, it only checks for 10.11.12.4:2181 and >>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is >>> where >>> >>> the >>> >>>> running server is. >>> >>>> >>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on >>> 07/21/2016 >>> >>>> 16:24 GMT >>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >>> >>> host.name >>> >>>> =w-rli09-ben >>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:java.version=1.8.0_11 >>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:java.vendor=Oracle Corporation >>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre >>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> >>> >>> >>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> >>> >>> >>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:java.io.tmpdir=/tmp >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:java.compiler=<NA> >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >>> >>> os.name >>> >>>> =Linux >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:os.arch=amd64 >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:os.version=3.8.0-44-generic >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >>> >>> user.name >>> >>>> =benji >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:user.home=/home/benji >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:os.memory.free=376MB >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:os.memory.max=5358MB >>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>> >>>> environment:os.memory.total=378MB >>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client >>> >>>> connection, connectString=10.11.9.70:2181 >>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181 >>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 < >>> http://10.11.12.210:2181> >>> >>>> sessionTimeout=45000 >>> watcher=testzkclient.TestZkReadOnlyClient@5010be6* >>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - >>> jute.maxbuffer >>> >>> is >>> >>>> 4194304 >>> >>>> Successfully established the connection with ZooKeeper >>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] >>> INFO >>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to >>> server >>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. >>> Will >>> >>> not >>> >>>> attempt to authenticate using SASL (unknown error)* >>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN >>> >>> org.apache.zookeeper.ClientCnxn >>> >>>> - Session 0x0 for server null, unexpected error, closing socket >>> >>> connection >>> >>>> and attempting reconnect >>> >>>> java.net.ConnectException: Connection refused >>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> >>>> at >>> >>>> >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >>> >>>> at >>> >>>> >>> >>> >>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) >>> >>>> at >>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>> during >>> >>>> shutdown input >>> >>>> java.nio.channels.ClosedChannelException >>> >>>> at >>> >>>> >>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >>> >>>> at >>> >>>> >>> >>> >>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) >>> >>>> at >>> >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>> >>>> at >>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>> during >>> >>>> shutdown output >>> >>>> java.nio.channels.ClosedChannelException >>> >>>> at >>> >>>> >>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >>> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) >>> >>>> at >>> >>>> >>> >>> >>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) >>> >>>> at >>> >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>> >>>> at >>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181 >>> >)] >>> >>> INFO >>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to >>> server >>> >>>> 10.11.12.210/10.11.12.210:2181 < >>> http://10.11.12.210/10.11.12.210:2181>. >>> >>>> Will not attempt to authenticate using SASL (unknown error)* >>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN >>> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >>> >>> unexpected >>> >>>> error, closing socket connection and attempting reconnect >>> >>>> java.net.ConnectException: Connection refused >>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> >>>> at >>> >>>> >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >>> >>>> at >>> >>>> >>> >>> >>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) >>> >>>> at >>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>> during >>> >>>> shutdown input >>> >>>> java.nio.channels.ClosedChannelException >>> >>>> at >>> >>>> >>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >>> >>>> at >>> >>>> >>> >>> >>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) >>> >>>> at >>> >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>> >>>> at >>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>> during >>> >>>> shutdown output >>> >>>> java.nio.channels.ClosedChannelException >>> >>>> at >>> >>>> >>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >>> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) >>> >>>> at >>> >>>> >>> >>> >>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) >>> >>>> at >>> >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>> >>>> at >>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>> >>>> *Exception in thread "main" >>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> >>>> KeeperErrorCode = ConnectionLoss for /* >>> >>>> at >>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >>> >>>> at >>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956) >>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985) >>> >>>> at >>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26) >>> >>>> at >>> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13) >>> >>>> >>> >>>> >>> >>>> Does anybody else have issues with the readonly mode? >>> >>>> >>> >>>> Thanks >>> >>>> Benjamin >>> >>>> >>> >>>> >>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton < >>> >>> [email protected]> >>> >>>> wrote: >>> >>>> >>> >>>>> Very interesting, thank you Flavio. >>> >>>>> >>> >>>>> That should not happen because I have actually never started (even >>> >>> never >>> >>>>> created) the other servers. This server is the first one the client >>> >>> could >>> >>>>> have possibly ever seen. Maybe the server has to have been in a >>> >>> functioning >>> >>>>> ensemble at least once before being able to be read only able? >>> >>>>> I will recheck all my setup and do it again. >>> >>>>> Thanks again, >>> >>>>> Ben >>> >>>>> >>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <[email protected]> >>> >>> wrote: >>> >>>>> >>> >>>>>> From the logs, it looks like the server the client is trying to >>> >>> connect >>> >>>>>> to is behind compared to last zxid the client has seen: >>> >>>>>> >>> >>>>>> Refusing session request for client /10.11.12.4:49237 as it >>> >>> has >>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server >>> >>>>>> >>> >>>>>> I believe the behavior we currently have is that a client can >>> connect >>> >>> to >>> >>>>>> an RO server, but the server must have a view at least as recent >>> as >>> >>> the >>> >>>>>> server. >>> >>>>>> >>> >>>>>> -Flavio >>> >>>>>> >>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton < >>> [email protected]> >>> >>>>>> wrote: >>> >>>>>>> >>> >>>>>>> Hello, >>> >>>>>>> >>> >>>>>>> I'm having problems using ZK in read-only mode. >>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure >>> the >>> >>>>>> installation to be in a 3 nodes ensemble. >>> >>>>>>> I start this node only (QA-E8WIN11) with >>> -Dreadonlymode.enabled=true. >>> >>>>>>> >>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly: >>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO >>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started >>> >>>>>>> >>> >>>>>>> Then I make a read-only enabled client: >>> >>>>>>> >>> >>>>>>> ZooKeeper zk = new >>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", >>> 45000, >>> >>> this, >>> >>>>>> true); >>> >>>>>>> System.out.println("Successfully established the connection >>> >>>>>> with ZooKeeper"); >>> >>>>>>> >>> >>>>>>> zk.getData("/", false, null); >>> >>>>>>> System.out.println("Done."); >>> >>>>>>> >>> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not. >>> >>>>>>> >>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works. >>> >>>>>>> But when it picks a different node, it fails with: >>> >>>>>>> >>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating >>> client >>> >>>>>> connection, >>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181 >>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145 >>> >>>>>>> Successfully established the connection with ZooKeeper >>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening >>> socket >>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 < >>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate using >>> >>> SASL >>> >>>>>> (unknown error) >>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client >>> session >>> >>>>>> timed out, have not heard from server in 15001ms for sessionid >>> 0x0, >>> >>> closing >>> >>>>>> socket connection and attempting reconnect >>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening >>> socket >>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 < >>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate using >>> >>> SASL >>> >>>>>> (unknown error) >>> >>>>>>> Exception in thread "main" >>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> >>>>>> KeeperErrorCode = ConnectionLoss for / >>> >>>>>>> at >>> >>>>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >>> >>>>>>> at >>> >>>>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753) >>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782) >>> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26) >>> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16) >>> >>>>>>> >>> >>>>>>> I'm attaching all the logs. The server and the client are on the >>> same >>> >>>>>> machine QA-E8WIN11. >>> >>>>>>> >>> >>>>>>> Any help would be greatly appreciated! >>> >>>>>>> >>> >>>>>>> Thanks, >>> >>>>>>> Benjamin Jaton >>> >>>>>>> <zkclient.log><zkserver.log> >>> >>>>>> >>> >>>>>> >>> >>>>> >>> >>> >>> >>> >>> >> >>> >>> >> >
