Sorry again I'm having reading issues ignore that comment as I see flavio already answered it
On Jul 29, 2016 5:25 PM, "Camille Fournier" <[email protected]> wrote: > Update, I was confused by races of my own doing. Was this client > previously connected when it failed the read only check? The server log is > rejecting it because the client is ahead in zxid in the first set of logs. > The other two servers are totally down and it's a new client when you see > this issue? > > On Jul 29, 2016 3:07 PM, "Camille Fournier" <[email protected]> wrote: > >> Ok yeah I think this is reproducible and a bug in the client connection >> read-only logic. >> >> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <[email protected]> >> wrote: >> >>> I'm looking at the readonly mode code right now and it appears that the >>> only way to set readonly mode is a global system property which means that >>> the tests for this are only testing across 3 servers, all of which have >>> readonly mode set. So, this MAY be a bug, but what a pain to figure out how >>> to reproduce. >>> >>> Global system properties: not even once. >>> >>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <[email protected]> >>> wrote: >>> >>>> Not that I'm aware of... >>>> >>>> -Flavio >>>> >>>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <[email protected]> >>>> wrote: >>>> > >>>> > Hi Flavio, >>>> > >>>> > The bug / patch is only for the C client, does this issue affect the >>>> java >>>> > client as well? >>>> > >>>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton < >>>> [email protected]> >>>> > wrote: >>>> > >>>> >> I will definitely try and I'll post an update, thanks again Flavio. >>>> >> >>>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <[email protected]> >>>> wrote: >>>> >> >>>> >>> I think you're hitting this: >>>> >>> >>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 < >>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466> >>>> >>> >>>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch. >>>> >>> >>>> >>> -Flavio >>>> >>> >>>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <[email protected] >>>> > >>>> >>> wrote: >>>> >>>> >>>> >>>> OK so I did another fresh install this time using 3.5.2-alpha, and >>>> the >>>> >>> read >>>> >>>> only client doesn't seem to try all the hosts available. >>>> >>>> >>>> >>>> I create my client with: >>>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181, >>>> >>>> 10.11.12.210:2181", 45000, this, true); >>>> >>>> >>>> >>>> In the following output, it only checks for 10.11.12.4:2181 and >>>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is >>>> where >>>> >>> the >>>> >>>> running server is. >>>> >>>> >>>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on >>>> 07/21/2016 >>>> >>>> 16:24 GMT >>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >>>> >>> host.name >>>> >>>> =w-rli09-ben >>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:java.version=1.8.0_11 >>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:java.vendor=Oracle Corporation >>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre >>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> >>>> >>> >>>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> >>>> >>> >>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:java.io.tmpdir=/tmp >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:java.compiler=<NA> >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >>>> >>> os.name >>>> >>>> =Linux >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:os.arch=amd64 >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:os.version=3.8.0-44-generic >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >>>> >>> user.name >>>> >>>> =benji >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:user.home=/home/benji >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:os.memory.free=376MB >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:os.memory.max=5358MB >>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >>>> >>>> environment:os.memory.total=378MB >>>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client >>>> >>>> connection, connectString=10.11.9.70:2181 >>>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181 >>>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 < >>>> http://10.11.12.210:2181> >>>> >>>> sessionTimeout=45000 >>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6* >>>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - >>>> jute.maxbuffer >>>> >>> is >>>> >>>> 4194304 >>>> >>>> Successfully established the connection with ZooKeeper >>>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] >>>> INFO >>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to >>>> server >>>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. >>>> Will >>>> >>> not >>>> >>>> attempt to authenticate using SASL (unknown error)* >>>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN >>>> >>> org.apache.zookeeper.ClientCnxn >>>> >>>> - Session 0x0 for server null, unexpected error, closing socket >>>> >>> connection >>>> >>>> and attempting reconnect >>>> >>>> java.net.ConnectException: Connection refused >>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>>> >>>> at >>>> >>>> >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >>>> >>>> at >>>> >>>> >>>> >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) >>>> >>>> at >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG >>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>>> during >>>> >>>> shutdown input >>>> >>>> java.nio.channels.ClosedChannelException >>>> >>>> at >>>> >>>> >>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >>>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >>>> >>>> at >>>> >>>> >>>> >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) >>>> >>>> at >>>> >>>> >>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>>> >>>> at >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG >>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>>> during >>>> >>>> shutdown output >>>> >>>> java.nio.channels.ClosedChannelException >>>> >>>> at >>>> >>>> >>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >>>> >>>> at >>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) >>>> >>>> at >>>> >>>> >>>> >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) >>>> >>>> at >>>> >>>> >>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>>> >>>> at >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181 >>>> >)] >>>> >>> INFO >>>> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to >>>> server >>>> >>>> 10.11.12.210/10.11.12.210:2181 < >>>> http://10.11.12.210/10.11.12.210:2181>. >>>> >>>> Will not attempt to authenticate using SASL (unknown error)* >>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN >>>> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >>>> >>> unexpected >>>> >>>> error, closing socket connection and attempting reconnect >>>> >>>> java.net.ConnectException: Connection refused >>>> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>>> >>>> at >>>> >>>> >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >>>> >>>> at >>>> >>>> >>>> >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) >>>> >>>> at >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG >>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>>> during >>>> >>>> shutdown input >>>> >>>> java.nio.channels.ClosedChannelException >>>> >>>> at >>>> >>>> >>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >>>> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >>>> >>>> at >>>> >>>> >>>> >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198) >>>> >>>> at >>>> >>>> >>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>>> >>>> at >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG >>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception >>>> during >>>> >>>> shutdown output >>>> >>>> java.nio.channels.ClosedChannelException >>>> >>>> at >>>> >>>> >>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >>>> >>>> at >>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410) >>>> >>>> at >>>> >>>> >>>> >>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205) >>>> >>>> at >>>> >>>> >>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >>>> >>>> at >>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >>>> >>>> *Exception in thread "main" >>>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> >>>> KeeperErrorCode = ConnectionLoss for /* >>>> >>>> at >>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >>>> >>>> at >>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956) >>>> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985) >>>> >>>> at >>>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26) >>>> >>>> at >>>> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13) >>>> >>>> >>>> >>>> >>>> >>>> Does anybody else have issues with the readonly mode? >>>> >>>> >>>> >>>> Thanks >>>> >>>> Benjamin >>>> >>>> >>>> >>>> >>>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton < >>>> >>> [email protected]> >>>> >>>> wrote: >>>> >>>> >>>> >>>>> Very interesting, thank you Flavio. >>>> >>>>> >>>> >>>>> That should not happen because I have actually never started (even >>>> >>> never >>>> >>>>> created) the other servers. This server is the first one the >>>> client >>>> >>> could >>>> >>>>> have possibly ever seen. Maybe the server has to have been in a >>>> >>> functioning >>>> >>>>> ensemble at least once before being able to be read only able? >>>> >>>>> I will recheck all my setup and do it again. >>>> >>>>> Thanks again, >>>> >>>>> Ben >>>> >>>>> >>>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <[email protected] >>>> > >>>> >>> wrote: >>>> >>>>> >>>> >>>>>> From the logs, it looks like the server the client is trying to >>>> >>> connect >>>> >>>>>> to is behind compared to last zxid the client has seen: >>>> >>>>>> >>>> >>>>>> Refusing session request for client /10.11.12.4:49237 as >>>> it >>>> >>> has >>>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server >>>> >>>>>> >>>> >>>>>> I believe the behavior we currently have is that a client can >>>> connect >>>> >>> to >>>> >>>>>> an RO server, but the server must have a view at least as recent >>>> as >>>> >>> the >>>> >>>>>> server. >>>> >>>>>> >>>> >>>>>> -Flavio >>>> >>>>>> >>>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton < >>>> [email protected]> >>>> >>>>>> wrote: >>>> >>>>>>> >>>> >>>>>>> Hello, >>>> >>>>>>> >>>> >>>>>>> I'm having problems using ZK in read-only mode. >>>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure >>>> the >>>> >>>>>> installation to be in a 3 nodes ensemble. >>>> >>>>>>> I start this node only (QA-E8WIN11) with >>>> -Dreadonlymode.enabled=true. >>>> >>>>>>> >>>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly: >>>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO >>>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started >>>> >>>>>>> >>>> >>>>>>> Then I make a read-only enabled client: >>>> >>>>>>> >>>> >>>>>>> ZooKeeper zk = new >>>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", >>>> 45000, >>>> >>> this, >>>> >>>>>> true); >>>> >>>>>>> System.out.println("Successfully established the >>>> connection >>>> >>>>>> with ZooKeeper"); >>>> >>>>>>> >>>> >>>>>>> zk.getData("/", false, null); >>>> >>>>>>> System.out.println("Done."); >>>> >>>>>>> >>>> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not. >>>> >>>>>>> >>>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works. >>>> >>>>>>> But when it picks a different node, it fails with: >>>> >>>>>>> >>>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating >>>> client >>>> >>>>>> connection, >>>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181 >>>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145 >>>> >>>>>>> Successfully established the connection with ZooKeeper >>>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening >>>> socket >>>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 < >>>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate >>>> using >>>> >>> SASL >>>> >>>>>> (unknown error) >>>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client >>>> session >>>> >>>>>> timed out, have not heard from server in 15001ms for sessionid >>>> 0x0, >>>> >>> closing >>>> >>>>>> socket connection and attempting reconnect >>>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening >>>> socket >>>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 < >>>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate >>>> using >>>> >>> SASL >>>> >>>>>> (unknown error) >>>> >>>>>>> Exception in thread "main" >>>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> >>>>>> KeeperErrorCode = ConnectionLoss for / >>>> >>>>>>> at >>>> >>>>>> >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >>>> >>>>>>> at >>>> >>>>>> >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >>>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753) >>>> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782) >>>> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26) >>>> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16) >>>> >>>>>>> >>>> >>>>>>> I'm attaching all the logs. The server and the client are on >>>> the same >>>> >>>>>> machine QA-E8WIN11. >>>> >>>>>>> >>>> >>>>>>> Any help would be greatly appreciated! >>>> >>>>>>> >>>> >>>>>>> Thanks, >>>> >>>>>>> Benjamin Jaton >>>> >>>>>>> <zkclient.log><zkserver.log> >>>> >>>>>> >>>> >>>>>> >>>> >>>>> >>>> >>> >>>> >>> >>>> >> >>>> >>>> >>> >>
