Just to clarify one thing though in the server logs from the fresh install is the new read only server reporting anything?
On Jul 29, 2016 5:26 PM, "Camille Fournier" <[email protected]> wrote: > Sorry again I'm having reading issues ignore that comment as I see flavio > already answered it > > On Jul 29, 2016 5:25 PM, "Camille Fournier" <[email protected]> wrote: > >> Update, I was confused by races of my own doing. Was this client >> previously connected when it failed the read only check? The server log is >> rejecting it because the client is ahead in zxid in the first set of logs. >> The other two servers are totally down and it's a new client when you see >> this issue? >> >> On Jul 29, 2016 3:07 PM, "Camille Fournier" <[email protected]> wrote: >> >>> Ok yeah I think this is reproducible and a bug in the client connection >>> read-only logic. >>> >>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <[email protected]> >>> wrote: >>> >>>> I'm looking at the readonly mode code right now and it appears that the >>>> only way to set readonly mode is a global system property which means that >>>> the tests for this are only testing across 3 servers, all of which have >>>> readonly mode set. So, this MAY be a bug, but what a pain to figure out how >>>> to reproduce. >>>> >>>> Global system properties: not even once. >>>> >>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <[email protected]> >>>> wrote: >>>> >>>>> Not that I'm aware of... >>>>> >>>>> -Flavio >>>>> >>>>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <[email protected]> >>>>> wrote: >>>>> > >>>>> > Hi Flavio, >>>>> > >>>>> > The bug / patch is only for the C client, does this issue affect the >>>>> java >>>>> > client as well? >>>>> > >>>>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton < >>>>> [email protected]> >>>>> > wrote: >>>>> > >>>>> >> I will definitely try and I'll post an update, thanks again Flavio. >>>>> >> >>>>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <[email protected]> >>>>> wrote: >>>>> >> >>>>> >>> I think you're hitting this: >>>>> >>> >>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 < >>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466> >>>>> >>> >>>>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch. >>>>> >>> >>>>> >>> -Flavio >>>>> >>> >>>>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton < >>>>> [email protected]> >>>>> >>> wrote: >>>>> >>>> >>>>> >>>> OK so I did another fresh install this time using 3.5.2-alpha, >>>>> and the >>>>> >>> read >>>>> >>>> only client doesn't seem to try all the hosts available. >>>>> >>>> >>>>> >>>> I create my client with: >>>>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181, >>>>> >>>> 10.11.12.210:2181", 45000, this, true); >>>>> >>>> >>>>> >>>> In the following output, it only checks for 10.11.12.4:2181 and >>>>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is >>>>> where >>>>> >>> the >>>>> >>>> running server is. >>>>> >>>> >>>>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on >>>>> 07/21/2016 >>>>> >>>> 16:24 GMT >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> environment: >>>>> >>> host.name >>>>> >>>> =w-rli09-ben >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:java.version=1.8.0_11 >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:java.vendor=Oracle Corporation >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre >>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> >>>>> >>> >>>>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> >>>>> >>> >>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:java.io.tmpdir=/tmp >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:java.compiler=<NA> >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> environment: >>>>> >>> os.name >>>>> >>>> =Linux >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:os.arch=amd64 >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:os.version=3.8.0-44-generic >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> environment: >>>>> >>> user.name >>>>> >>>> =benji >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:user.home=/home/benji >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:os.memory.free=376MB >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:os.memory.max=5358MB >>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>>> >>>> environment:os.memory.total=378MB >>>>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client >>>>> >>>> connection, connectString=10.11.9.70:2181 >>>>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181 >>>>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 < >>>>> http://10.11.12.210:2181> >>>>> >>>> sessionTimeout=45000 >>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6* >>>>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - >>>>> jute.maxbuffer >>>>> >>> is >>>>> >>>> 4194304 >>>>> >>>> Successfully established the connection with ZooKeeper >>>>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] >>>>> INFO >>>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to >>>>> server >>>>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. >>>>> Will >>>>> >>> not >>>>> >>>> attempt to authenticate using SASL (unknown error)* >>>>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN >>>>> >>> org.apache.zookeeper.ClientCnxn >>>>> >>>> - Session 0x0 for server null, unexpected error, closing socket >>>>> >>> connection >>>>> >>>> and attempting reconnect >>>>> >>>> java.net.ConnectException: Connection refused >>>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>>>> >>>> at >>>>> >>>> >>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >>>>> >>>> at >>>>> >>>> >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) >>>>> >>>> at >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG >>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>>>> during >>>>> >>>> shutdown input >>>>> >>>> java.nio.channels.ClosedChannelException >>>>> >>>> at >>>>> >>>> >>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >>>>> >>>> at >>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >>>>> >>>> at >>>>> >>>> >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) >>>>> >>>> at >>>>> >>>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>>>> >>>> at >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG >>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>>>> during >>>>> >>>> shutdown output >>>>> >>>> java.nio.channels.ClosedChannelException >>>>> >>>> at >>>>> >>>> >>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >>>>> >>>> at >>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) >>>>> >>>> at >>>>> >>>> >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) >>>>> >>>> at >>>>> >>>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>>>> >>>> at >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>>>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181 >>>>> >)] >>>>> >>> INFO >>>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to >>>>> server >>>>> >>>> 10.11.12.210/10.11.12.210:2181 < >>>>> http://10.11.12.210/10.11.12.210:2181>. >>>>> >>>> Will not attempt to authenticate using SASL (unknown error)* >>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN >>>>> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >>>>> >>> unexpected >>>>> >>>> error, closing socket connection and attempting reconnect >>>>> >>>> java.net.ConnectException: Connection refused >>>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>>>> >>>> at >>>>> >>>> >>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >>>>> >>>> at >>>>> >>>> >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) >>>>> >>>> at >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG >>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>>>> during >>>>> >>>> shutdown input >>>>> >>>> java.nio.channels.ClosedChannelException >>>>> >>>> at >>>>> >>>> >>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >>>>> >>>> at >>>>> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >>>>> >>>> at >>>>> >>>> >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) >>>>> >>>> at >>>>> >>>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>>>> >>>> at >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG >>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>>>> during >>>>> >>>> shutdown output >>>>> >>>> java.nio.channels.ClosedChannelException >>>>> >>>> at >>>>> >>>> >>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >>>>> >>>> at >>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) >>>>> >>>> at >>>>> >>>> >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) >>>>> >>>> at >>>>> >>>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>>>> >>>> at >>>>> >>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>>>> >>>> *Exception in thread "main" >>>>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> >>>> KeeperErrorCode = ConnectionLoss for /* >>>>> >>>> at >>>>> >>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >>>>> >>>> at >>>>> >>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >>>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956) >>>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985) >>>>> >>>> at >>>>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26) >>>>> >>>> at >>>>> >>> >>>>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13) >>>>> >>>> >>>>> >>>> >>>>> >>>> Does anybody else have issues with the readonly mode? >>>>> >>>> >>>>> >>>> Thanks >>>>> >>>> Benjamin >>>>> >>>> >>>>> >>>> >>>>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton < >>>>> >>> [email protected]> >>>>> >>>> wrote: >>>>> >>>> >>>>> >>>>> Very interesting, thank you Flavio. >>>>> >>>>> >>>>> >>>>> That should not happen because I have actually never started >>>>> (even >>>>> >>> never >>>>> >>>>> created) the other servers. This server is the first one the >>>>> client >>>>> >>> could >>>>> >>>>> have possibly ever seen. Maybe the server has to have been in a >>>>> >>> functioning >>>>> >>>>> ensemble at least once before being able to be read only able? >>>>> >>>>> I will recheck all my setup and do it again. >>>>> >>>>> Thanks again, >>>>> >>>>> Ben >>>>> >>>>> >>>>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira < >>>>> [email protected]> >>>>> >>> wrote: >>>>> >>>>> >>>>> >>>>>> From the logs, it looks like the server the client is trying to >>>>> >>> connect >>>>> >>>>>> to is behind compared to last zxid the client has seen: >>>>> >>>>>> >>>>> >>>>>> Refusing session request for client /10.11.12.4:49237 as >>>>> it >>>>> >>> has >>>>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another >>>>> server >>>>> >>>>>> >>>>> >>>>>> I believe the behavior we currently have is that a client can >>>>> connect >>>>> >>> to >>>>> >>>>>> an RO server, but the server must have a view at least as >>>>> recent as >>>>> >>> the >>>>> >>>>>> server. >>>>> >>>>>> >>>>> >>>>>> -Flavio >>>>> >>>>>> >>>>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton < >>>>> [email protected]> >>>>> >>>>>> wrote: >>>>> >>>>>>> >>>>> >>>>>>> Hello, >>>>> >>>>>>> >>>>> >>>>>>> I'm having problems using ZK in read-only mode. >>>>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure >>>>> the >>>>> >>>>>> installation to be in a 3 nodes ensemble. >>>>> >>>>>>> I start this node only (QA-E8WIN11) with >>>>> -Dreadonlymode.enabled=true. >>>>> >>>>>>> >>>>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly: >>>>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO >>>>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server >>>>> started >>>>> >>>>>>> >>>>> >>>>>>> Then I make a read-only enabled client: >>>>> >>>>>>> >>>>> >>>>>>> ZooKeeper zk = new >>>>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", >>>>> 45000, >>>>> >>> this, >>>>> >>>>>> true); >>>>> >>>>>>> System.out.println("Successfully established the >>>>> connection >>>>> >>>>>> with ZooKeeper"); >>>>> >>>>>>> >>>>> >>>>>>> zk.getData("/", false, null); >>>>> >>>>>>> System.out.println("Done."); >>>>> >>>>>>> >>>>> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not. >>>>> >>>>>>> >>>>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works. >>>>> >>>>>>> But when it picks a different node, it fails with: >>>>> >>>>>>> >>>>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating >>>>> client >>>>> >>>>>> connection, >>>>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181 >>>>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145 >>>>> >>>>>>> Successfully established the connection with ZooKeeper >>>>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening >>>>> socket >>>>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 < >>>>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate >>>>> using >>>>> >>> SASL >>>>> >>>>>> (unknown error) >>>>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client >>>>> session >>>>> >>>>>> timed out, have not heard from server in 15001ms for sessionid >>>>> 0x0, >>>>> >>> closing >>>>> >>>>>> socket connection and attempting reconnect >>>>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening >>>>> socket >>>>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 < >>>>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate >>>>> using >>>>> >>> SASL >>>>> >>>>>> (unknown error) >>>>> >>>>>>> Exception in thread "main" >>>>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>>> >>>>>> KeeperErrorCode = ConnectionLoss for / >>>>> >>>>>>> at >>>>> >>>>>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >>>>> >>>>>>> at >>>>> >>>>>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >>>>> >>>>>>> at >>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753) >>>>> >>>>>>> at >>>>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782) >>>>> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26) >>>>> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16) >>>>> >>>>>>> >>>>> >>>>>>> I'm attaching all the logs. The server and the client are on >>>>> the same >>>>> >>>>>> machine QA-E8WIN11. >>>>> >>>>>>> >>>>> >>>>>>> Any help would be greatly appreciated! >>>>> >>>>>>> >>>>> >>>>>>> Thanks, >>>>> >>>>>>> Benjamin Jaton >>>>> >>>>>>> <zkclient.log><zkserver.log> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>> >>>>> >>> >>>>> >>> >>>>> >> >>>>> >>>>> >>>> >>>
