I was able to reproduce using your testing method and get it working locally... I must have a firewall port or something in the way. Thanks, Eric
On Thu, Dec 30, 2021 at 2:24 PM Alexander Shraer <[email protected]> wrote: > I used master. But as long as you’re using 3.5 and on you should be fine. > I’m not sure when the acl option was added but the standaloneEnabled and > reconfig features were in 3.5.0 > > On Thu, Dec 30, 2021 at 11:51 AM Eric Edgar > <[email protected]> wrote: > > > This is great. I should confirm what zk version you are using for your > > tests. > > Thanks, > > Eric > > > > On Thu, Dec 30, 2021 at 1:10 PM Alexander Shraer <[email protected]> > > wrote: > > > > > The reconfig is in process means something failed during > reconfiguration > > > and it couldn't complete. Perhaps the new server disconnected in the > > middle > > > and never came back up. Notice that the second server's config file > gets > > > overwritten after it connects to the leader, and if it reboots at this > > > stage it won't be able to connect again without you manually > overwriting > > > its config file again (since in the server's config server 2 is not > part > > of > > > the ensemble). > > > > > > I checked it locally (running both servers on my laptop), and it > worked. > > > Perhaps start from that ? > > > > > > Like you said, I disabled acl by adding > > > > > > "-Dzookeeper.skipACL=yes" > > > > > > Here's the first server's config file: conf/zoo_replicated1.cfg > > > > > > dataDir=/Users/shralex/my-zookeeper/zookeeper1 > > > > > > syncLimit=2 > > > > > > initLimit=5 > > > > > > tickTime=2000 > > > > > > clientPort=2791 > > > > > > reconfigEnabled=true > > > > > > standaloneEnabled=false > > > > > > server.1=localhost:2721:2731:participant;localhost:2791 > > > > > > The second server's: conf/zoo_replicated2.cfg > > > > > > ataDir=/Users/shralex/my-zookeeper/zookeeper2 > > > > > > syncLimit=2 > > > > > > initLimit=5 > > > > > > tickTime=2000 > > > > > > clientPort=2792 > > > > > > reconfigEnabled=true > > > > > > standaloneEnabled=false > > > > > > server.1=localhost:2721:2731:participant;localhost:2791 > > > > > > server.2=localhost:2741:2751:participant;localhost:2792 > > > > > > create 2 directories for the servers: zookeeper1 and zookeeper2 and > > create > > > myid files in each > > > > > > echo 1 > zookeeper1/myid > > > > > > echo 2 > zookeeper2/myid > > > > > > I find it easier for debugging to allow zkServer.sh to log to stdout. > You > > > can do this by changing zkServer.sh: > > > - change nohup "$JAVA" to just "$JAVA" > > > - remove " > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null" > > > > > > In two shells start both servers by > > > > > > export ZOOCFG=zoo_replicated1.cfg (change for server 2) > > > > > > ./bin/zkServer.sh start > > > > > > In a third shell I start the client by connecting it to server 2 as you > > did > > > > > > ./bin/zkCli.sh -server 127.0.0.1:2792 > > > > > > I run the following in the shell: > > > > > > [zk: 127.0.0.1:2792(CONNECTED) 2] config > > > > > > server.1=localhost:2721:2731:participant;localhost:2791 > > > > > > version=100000000 > > > > > > [zk: 127.0.0.1:2792(CONNECTED) 2] reconfig -add > > > "server.2=localhost:2741:2751:participant;localhost:2792" > > > > > > Committed new configuration: > > > > > > server.1=localhost:2721:2731:participant;localhost:2791 > > > > > > server.2=localhost:2741:2751:participant;localhost:2792 > > > > > > version=200000003 > > > > > > On Thu, Dec 30, 2021 at 10:47 AM Eric Edgar > > > <[email protected]> wrote: > > > > > > > I am a little closer I think. I disabled auth for testing using the > > > server > > > > flags .. but now I am getting a different error that the reconfig is > in > > > > process and I see a zookeeper.dynamic.next file on both servers but > > > nothing > > > > happens after that. > > > > What would cause that file to not be merged into a new cfg. > > > > Eric > > > > > > > > On Thu, Dec 30, 2021 at 11:47 AM Eric Edgar < > > [email protected]> > > > > wrote: > > > > > > > > > Alex, > > > > > so I have 2 nodes .. the first has itself in the dynamic list with > an > > > id > > > > > of 1. > > > > > server.1=10.1.1.104:2888:3888:participant;0.0.0.0:2181 > > > > > > > > > > I have brought the second node up with an id of 2 > > > > > server.1=10.1.1.104:2888:3888:participant;0.0.0.0:2181 > > > > > server.2=10.1.1.40:2888:3888:participant;2181 > > > > > > > > > > then i am trying to run from the second node. zkCli.sh -server > > > > 10.1.1.104 > > > > > reconfig -add "server.2=10.1.1.40:2888:3888:participant;2181" > > > > > > > > > > > > > > > > > > > > I get this error on the first server > > > > > 2021-12-30 17:37:02,880 [myid:1] - INFO [ProcessThread(sid:1 > > > > > cport:-1)::PrepRequestProcessor@461] - Incremental reconfig > > > > > 2021-12-30 17:37:02,880 [myid:1] - WARN [ProcessThread(sid:1 > > > > > cport:-1)::PrepRequestProcessor@532] - Reconfig failed - there > must > > > be a > > > > > connected and synced quorum in new configuration > > > > > 2021-12-30 17:37:02,880 [myid:1] - INFO [ProcessThread(sid:1 > > > > > cport:-1)::PrepRequestProcessor@935] - Got user-level > > KeeperException > > > > > when processing sessionid:0x1002dfe65610014 type:reconfig cxid:0x1 > > > > > zxid:0x1600000033 txntype:-1 reqpath:n > > > > > > > > > > > > > > > on the second server issuing the reconfig command I get this error > > > > > No quorum of new config is connected and up-to-date with the leader > > of > > > > > last commmitted config - try invoking reconfiguration after new > > servers > > > > are > > > > > connected and synced > > > > > > > > > > I have not set any security at this point. > > > > > > > > > > I am not sure what I am missing at this point, assuming I don't > need > > 2 > > > > > nodes fully clustered in advance as mentioned by Chris. > > > > > > > > > > Thanks, > > > > > Eric > > > > > > > > > > On Thu, Dec 30, 2021 at 11:03 AM Alexander Shraer < > [email protected] > > > > > > > > wrote: > > > > > > > > > >> This is already possible, since the 3.5.0 release: > > > > >> > > > > >> > > > > > > > > > > https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html#sc_reconfig_standaloneEnabled > > > > >> > > > > >> After your single node is up and running, you can connect other > > nodes > > > to > > > > >> it > > > > >> as described in the reconfig manual. See "Adding servers" in the > > link > > > > >> above. > > > > >> Essentially, you need to specify the new server's initial config > > files > > > > so > > > > >> that they can find some existing server and start syncing data. > > Once a > > > > >> quorum > > > > >> of the new config is up to date, you can invoke the reconfig > command > > > to > > > > >> officially make them part of the configuration. > > > > >> > > > > >> Thanks, > > > > >> Alex > > > > >> > > > > >> On Thu, Dec 30, 2021 at 8:57 AM Eric Edgar > > > > >> <[email protected]> wrote: > > > > >> > > > > >> > Also would it be possible to update the code for this edge case, > > eg > > > > if > > > > >> the > > > > >> > current quorum is 1, and you want to add a node then add a flag > > > > saying I > > > > >> > trust the single master and reconfigure itself into a 2 node > > > cluster? > > > > >> > Thanks, > > > > >> > Eric > > > > >> > > > > > >> > On Thu, Dec 30, 2021 at 10:49 AM Eric Edgar < > > > > [email protected] > > > > >> > > > > > >> > wrote: > > > > >> > > > > > >> > > Are there any examples with a k8 orchestrator or some sort of > > > docker > > > > >> init > > > > >> > > scripts handling the initial cluster configuration? > > > > >> > > Thanks, > > > > >> > > Eric > > > > >> > > > > > > >> > > On Thu, Dec 30, 2021 at 9:44 AM Chris T. < > [email protected]> > > > > >> wrote: > > > > >> > > > > > > >> > >> If you want to run a zookeeper cluster you have to start with > > at > > > > >> least 2 > > > > >> > >> members. From there you can scale up with the dynamic > reconfig > > > > >> commands. > > > > >> > >> Regards > > > > >> > >> Chris > > > > >> > >> > > > > >> > >> On 30 December 2021 16:40:40 Eric Edgar > > > > >> > >> <[email protected]> wrote: > > > > >> > >> > > > > >> > >> > I am experimenting with zk and the reconfig feature and > > trying > > > to > > > > >> > >> > understand if I can start a single zk node and then > > > > >> reconfig/bootstrap > > > > >> > >> the > > > > >> > >> > other 2 nodes into the ensemble. The reconfig command is > > > > throwing > > > > >> an > > > > >> > >> error > > > > >> > >> > that there isn't a quorum yet. Is this line of thinking > > > > >> possible? or > > > > >> > >> do I > > > > >> > >> > need to setup the first 3 nodes manually the first time? > > > > >> > >> > I am basing this experiment off of this web page. > > > > >> > >> > > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > https://blog.container-solutions.com/dynamic-zookeeper-cluster-with-docker > > > > >> > >> > > > > > >> > >> > /opt/zookeeper/zookeeper/bin/zkCli.sh -server > > 10.1.1.104:2181 > > > > >> > reconfig > > > > >> > >> -add > > > > >> > >> > "server.2=10.1.1.40:2888:3888:participant;2181" > > > > >> > >> > No quorum of new config is connected and up-to-date with > the > > > > >> leader of > > > > >> > >> last > > > > >> > >> > commmitted config - try invoking reconfiguration after new > > > > servers > > > > >> are > > > > >> > >> > connected and synced > > > > >> > >> > > > > > >> > >> > /opt/zookeeper/zookeeper/bin/zkCli.sh -server > > 10.1.1.104:2181 > > > > >> config > > > > >> > >> > server.1=10.1.1.104:2888:3888:participant;0.0.0.0:2181 > > > > >> > >> > > > > > >> > >> > cat ./zoo.cfg > > > > >> > >> > autopurge.purgeInterval=1 > > > > >> > >> > initLimit=10 > > > > >> > >> > syncLimit=5 > > > > >> > >> > autopurge.snapRetainCount=6 > > > > >> > >> > tickTime=2000 > > > > >> > >> > dataDir=/mnt/zookeeper/data > > > > >> > >> > reconfigEnabled=true > > > > >> > >> > standaloneEnabled=false > > > > >> > >> > > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > dynamicConfigFile=/opt/zookeeper/zookeeper/conf/zoo.cfg.dynamic.1600000000 > > > > >> > >> > > > > > >> > >> > What is the best solution for an unattended bootstrap setup > > of > > > a > > > > >> new > > > > >> > >> > cluster from scratch? > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > This was something that we were able to accomplish with > > > exhibitor > > > > >> on > > > > >> > >> older > > > > >> > >> > versions of zookeeper in the past. > > > > >> > >> > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > > > > > >
