Re: Re: Any change from 3.6.3 -> 3.6.4 would cause hostname unresolved issue?
Paolo, Il Lun 19 Giu 2023, 16:43 Paolo Patierno ha scritto: > Hi all, > We were able to overcome the binding issue by setting > quorumListenOnAllIPs=true but from there we are getting a new issue that is > preventing leader election completion on first start-up. > > Getting the log of the current ZooKeeper leader (ID=3) we see the > following. > (Starting with ** you can see some additional logs added to > org.apache.zookeeper.server.quorum.Leader#getDesignatedLeader in order to > get more information.) > > 2023-06-19 12:32:51,990 INFO Have quorum of supporters, sids: [[1, 3],[1, > 3]]; starting up and setting last processed zxid: 0x1 > (org.apache.zookeeper.server.quorum.Leader) > [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2023-06-19 12:32:51,990 INFO ** > > newQVAcksetPair.getQuorumVerifier().getVotingMembers().get(self.getId()).addr > = my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/ > 172.17.0.6:2888 (org.apache.zookeeper.server.quorum.Leader) > [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2023-06-19 12:32:51,990 INFO ** self.getQuorumAddress() = > > my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/:2888 > (org.apache.zookeeper.server.quorum.Leader) > [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2023-06-19 12:32:51,992 INFO ** qs.addr > my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/ > 172.17.0.6:2888, qs.electionAddr > my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/ > 172.17.0.6:3888, qs.clientAddr/127.0.0.1:12181 > (org.apache.zookeeper.server.quorum.QuorumPeer) > [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2023-06-19 12:32:51,992 DEBUG zookeeper > (org.apache.zookeeper.common.PathTrie) > [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2023-06-19 12:32:51,993 WARN Restarting Leader Election > (org.apache.zookeeper.server.quorum.QuorumPeer) > [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > > So the leader is ZooKeeper with ID=3 and it was ACKed by the ZooKeeper node > ID=1. > As you can see we are in the Leader#startZkServer method, and because of > the reconfiguration enabled, the designatedLeader is processed. The problem > is that the Leader#getDesignatedLeader is not returning “self” as leader > but another one (ID=1), because of the difference in the quorum address. > From the above log, it’s not an actual difference in terms of addresses but > the self.getQuorumAddress() is returning an one (even if it’s > still the same hostname related to ZooKeeper-2 instance). This difference > causes the allowedToCommit=false, meanwhile the ZooKeeper-2 is still > reported as leader but it’s not able to commit, so prevents any requests > and the ZooKeeper ensemble gets stuck. > > 2023-06-19 12:32:51,996 WARN Suggested leader: 1 > (org.apache.zookeeper.server.quorum.QuorumPeer) > [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2023-06-19 12:32:51,996 WARN This leader is not the designated leader, it > will be initialized with allowedToCommit = false > (org.apache.zookeeper.server.quorum.Leader) > [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > > The overall issue could be related to DNS problems, with DNS records not > registered yet during pod initialization (where ZooKeeper is running on > Kubernetes). But we don’t understand why it’s not able to recover somehow. > > Instead of using the quorumListenOnAllIPs=true we also tried a different > approach by using the 0.0.0.0 address for the binding, so something like: > > # Zookeeper nodes configuration > server.1=0.0.0.0:2888:3888:participant;127.0.0.1:12181 > > server.2=my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.default.svc:2888:3888:participant; > 127.0.0.1:12181 > > server.3=my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc:2888:3888:participant; > 127.0.0.1:12181 > > This way, the self.getQuorumAddress() is not suffering the same problem, it > doesn’t return an address but always an actual one. No new > leader election is needed and everything works fine. > This is the release notes page for 3.6.4. https://zookeeper.apache.org/doc/r3.6.4/releasenotes.html As you are running on k8s, I guess you are using a statefulset, maybe with a service with ClusterIP? Is the readyness probe failing? In that case the dns name should not be available What are you using to perform the probe? Are you using the four letters words API ? (ruok) Enrico > On 2023/06/14 06:19:24 Szalay-Bekő Máté wrote: > > Interesting... > > > > I am not familiar with strimzi.io. > > Quickly checking the release notes, I don't see anything suspicious: > > https://zookeeper.apache.org/doc/r3.6.4/releasenotes.html > > Also, QuorumCnxManager was not changed for 2+ years on branch 3.6. > > > > Are you use the same java version and zookeeper config for 3.6.3 and > 3.6.4? > > Can you share the zookeeper config? > > > > Also: zookeeper 3.6 is
Re: Convert Standalone zookeeper to A 3 node Quorum
Indeed I went through this Admin guide for my version 3.5.8 and iam a bit confused with this attribute standaloneEnabled it doesn't seems to present as a part of Pristine/out of the box config but docs says its value is true by default, what does it exactly govern , more specific is this be part of my ensemble zk server config? PS thanks for the GitHub repo link Regards, Gaurav On Tue, 20 Jun, 2023, 01:04 Patrick Hunt, wrote: > It really depends on your requirements. You should read the admin docs for > insight and examples, start here: > https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_designing > > That said, I have a project here which I use for config generation/testing. > YMMV: > https://github.com/phunt/zkconf > > Patrick > > On Mon, Jun 19, 2023 at 11:32 AM Gaurav Pande > wrote: > > > Okay thanks, and could you share what's the standard config for ensemble > as > > example? I mean what all parameters it includes. > > > > Also do I need to add standaloneEnabled=false on these 2 new zk servers > > config file as a part of initial base ensemble config as by default this > > parameter value is true? So that Making 2 zk's run in replicated mode > > from starting (starting one at a time) > > > > Regards, > > Gaurav > > > > > > On Mon, 19 Jun, 2023, 22:03 Patrick Hunt, wrote: > > > > > On Mon, Jun 19, 2023 at 8:47 AM Gaurav Pande > > wrote: > > > > > > > Hi Patrick, > > > > > > > > Thanks for guidance here , based on below I only have presently 1 zk > > > node > > > > so if I provision 2 new VM and install same version of zookeeper on > > them > > > > should I start them as standalone zookeepers first and then make > > changes > > > to > > > > there server config file? > > > > > > > > > > > No, you specifically want to add them as part of the ensemble, one at a > > > time in sequence (see 2). > > > > > > > > > > Also what are the valid zk configs that I would need to add on these > 2 > > > new > > > > VM's zk server config file? > > > > > > > > > > > You'll need a regular ensemble config, not a standalone. > > > > > > Patrick > > > > > > > > > > Regards, > > > > Gaurav > > > > > > > > On Mon, 19 Jun, 2023, 21:06 Patrick Hunt, wrote: > > > > > > > > > Two ways to do it come to mind, which I've used in the past: > > > > > > > > > > 1) most straightforward is to "clone" the repos for the two new > > members > > > > of > > > > > the ensemble in their respective configs/datadirs. Just make sure > to > > > > update > > > > > the configs appropriately. Including the "myid" for each server. > Then > > > > > restart the ensemble and verify. > > > > > > > > > > 2) You can't go from 1->3 servers just by adding two new servers to > > the > > > > > ensemble as they may form a quorum on the "zero" zxid, rather than > > the > > > > zxid > > > > > of the existing member. Rather you would need to go from 1->2, with > > > > quorum, > > > > > and then from 2->3 with quorum. This will ensure that the true > state > > of > > > > the > > > > > original quorum is maintained (this is what we implemented for > > Cloudera > > > > > Manager to ensure proper functioning when increasing quorum size). > > > > > > > > > > Good luck, > > > > > > > > > > Patrick > > > > > > > > > > > > > > > On Mon, Jun 19, 2023 at 12:02 AM Gaurav Pande < > gaupand...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi tison, > > > > > > > > > > > > When you say stop services you mean existing standalone Zk > service > > > > right? > > > > > > , if that's the case then yes we can. But what's the process ? > Also > > > can > > > > > it > > > > > > be done without re-start I didn't know , can you share both > > > > > process/steps? > > > > > > > > > > > > On Mon, 19 Jun, 2023, 10:15 tison, wrote: > > > > > > > > > > > > > Can you stop the services for reconfig, or you need an online > > > > reconfig? > > > > > > > > > > > > > > Best, > > > > > > > tison. > > > > > > > > > > > > > > > > > > > > > Gaurav Pande 于2023年6月19日周一 11:22写道： > > > > > > > > > > > > > > > Hi Guys, > > > > > > > > > > > > > > > > Any help on this thread please? > > > > > > > > > > > > > > > > Regards, > > > > > > > > Gaurav > > > > > > > > > > > > > > > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, < > > gaupand...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello Guys, > > > > > > > > > > > > > > > > > > Iam new in this space, I wanted to know process/steps to > > > > convert a > > > > > > > > single > > > > > > > > > Zk node in standalone presently to a 3 node ZK quorum . > > > > > > > > > > > > > > > > > > What steps should I perform to add make my standalone Zk > > node a > > > > > > Quorum > > > > > > > of > > > > > > > > > 3 zk nodes? > > > > > > > > > > > > > > > > > > I am using ZK version 3.5.8 presently. > > > > > > > > > > > > > > > > > > Note - I have single Zookeeper(Standalone) and 3 Kafka > > brokers > > > at > > > > > > this > > > > > > > > > point. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Gaurav >
Re: Convert Standalone zookeeper to A 3 node Quorum
It really depends on your requirements. You should read the admin docs for insight and examples, start here: https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_designing That said, I have a project here which I use for config generation/testing. YMMV: https://github.com/phunt/zkconf Patrick On Mon, Jun 19, 2023 at 11:32 AM Gaurav Pande wrote: > Okay thanks, and could you share what's the standard config for ensemble as > example? I mean what all parameters it includes. > > Also do I need to add standaloneEnabled=false on these 2 new zk servers > config file as a part of initial base ensemble config as by default this > parameter value is true? So that Making 2 zk's run in replicated mode > from starting (starting one at a time) > > Regards, > Gaurav > > > On Mon, 19 Jun, 2023, 22:03 Patrick Hunt, wrote: > > > On Mon, Jun 19, 2023 at 8:47 AM Gaurav Pande > wrote: > > > > > Hi Patrick, > > > > > > Thanks for guidance here , based on below I only have presently 1 zk > > node > > > so if I provision 2 new VM and install same version of zookeeper on > them > > > should I start them as standalone zookeepers first and then make > changes > > to > > > there server config file? > > > > > > > > No, you specifically want to add them as part of the ensemble, one at a > > time in sequence (see 2). > > > > > > > Also what are the valid zk configs that I would need to add on these 2 > > new > > > VM's zk server config file? > > > > > > > > You'll need a regular ensemble config, not a standalone. > > > > Patrick > > > > > > > Regards, > > > Gaurav > > > > > > On Mon, 19 Jun, 2023, 21:06 Patrick Hunt, wrote: > > > > > > > Two ways to do it come to mind, which I've used in the past: > > > > > > > > 1) most straightforward is to "clone" the repos for the two new > members > > > of > > > > the ensemble in their respective configs/datadirs. Just make sure to > > > update > > > > the configs appropriately. Including the "myid" for each server. Then > > > > restart the ensemble and verify. > > > > > > > > 2) You can't go from 1->3 servers just by adding two new servers to > the > > > > ensemble as they may form a quorum on the "zero" zxid, rather than > the > > > zxid > > > > of the existing member. Rather you would need to go from 1->2, with > > > quorum, > > > > and then from 2->3 with quorum. This will ensure that the true state > of > > > the > > > > original quorum is maintained (this is what we implemented for > Cloudera > > > > Manager to ensure proper functioning when increasing quorum size). > > > > > > > > Good luck, > > > > > > > > Patrick > > > > > > > > > > > > On Mon, Jun 19, 2023 at 12:02 AM Gaurav Pande > > > > wrote: > > > > > > > > > Hi tison, > > > > > > > > > > When you say stop services you mean existing standalone Zk service > > > right? > > > > > , if that's the case then yes we can. But what's the process ? Also > > can > > > > it > > > > > be done without re-start I didn't know , can you share both > > > > process/steps? > > > > > > > > > > On Mon, 19 Jun, 2023, 10:15 tison, wrote: > > > > > > > > > > > Can you stop the services for reconfig, or you need an online > > > reconfig? > > > > > > > > > > > > Best, > > > > > > tison. > > > > > > > > > > > > > > > > > > Gaurav Pande 于2023年6月19日周一 11:22写道： > > > > > > > > > > > > > Hi Guys, > > > > > > > > > > > > > > Any help on this thread please? > > > > > > > > > > > > > > Regards, > > > > > > > Gaurav > > > > > > > > > > > > > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, < > gaupand...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hello Guys, > > > > > > > > > > > > > > > > Iam new in this space, I wanted to know process/steps to > > > convert a > > > > > > > single > > > > > > > > Zk node in standalone presently to a 3 node ZK quorum . > > > > > > > > > > > > > > > > What steps should I perform to add make my standalone Zk > node a > > > > > Quorum > > > > > > of > > > > > > > > 3 zk nodes? > > > > > > > > > > > > > > > > I am using ZK version 3.5.8 presently. > > > > > > > > > > > > > > > > Note - I have single Zookeeper(Standalone) and 3 Kafka > brokers > > at > > > > > this > > > > > > > > point. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Gaurav > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Convert Standalone zookeeper to A 3 node Quorum
Okay thanks, and could you share what's the standard config for ensemble as example? I mean what all parameters it includes. Also do I need to add standaloneEnabled=false on these 2 new zk servers config file as a part of initial base ensemble config as by default this parameter value is true? So that Making 2 zk's run in replicated mode from starting (starting one at a time) Regards, Gaurav On Mon, 19 Jun, 2023, 22:03 Patrick Hunt, wrote: > On Mon, Jun 19, 2023 at 8:47 AM Gaurav Pande wrote: > > > Hi Patrick, > > > > Thanks for guidance here , based on below I only have presently 1 zk > node > > so if I provision 2 new VM and install same version of zookeeper on them > > should I start them as standalone zookeepers first and then make changes > to > > there server config file? > > > > > No, you specifically want to add them as part of the ensemble, one at a > time in sequence (see 2). > > > > Also what are the valid zk configs that I would need to add on these 2 > new > > VM's zk server config file? > > > > > You'll need a regular ensemble config, not a standalone. > > Patrick > > > > Regards, > > Gaurav > > > > On Mon, 19 Jun, 2023, 21:06 Patrick Hunt, wrote: > > > > > Two ways to do it come to mind, which I've used in the past: > > > > > > 1) most straightforward is to "clone" the repos for the two new members > > of > > > the ensemble in their respective configs/datadirs. Just make sure to > > update > > > the configs appropriately. Including the "myid" for each server. Then > > > restart the ensemble and verify. > > > > > > 2) You can't go from 1->3 servers just by adding two new servers to the > > > ensemble as they may form a quorum on the "zero" zxid, rather than the > > zxid > > > of the existing member. Rather you would need to go from 1->2, with > > quorum, > > > and then from 2->3 with quorum. This will ensure that the true state of > > the > > > original quorum is maintained (this is what we implemented for Cloudera > > > Manager to ensure proper functioning when increasing quorum size). > > > > > > Good luck, > > > > > > Patrick > > > > > > > > > On Mon, Jun 19, 2023 at 12:02 AM Gaurav Pande > > > wrote: > > > > > > > Hi tison, > > > > > > > > When you say stop services you mean existing standalone Zk service > > right? > > > > , if that's the case then yes we can. But what's the process ? Also > can > > > it > > > > be done without re-start I didn't know , can you share both > > > process/steps? > > > > > > > > On Mon, 19 Jun, 2023, 10:15 tison, wrote: > > > > > > > > > Can you stop the services for reconfig, or you need an online > > reconfig? > > > > > > > > > > Best, > > > > > tison. > > > > > > > > > > > > > > > Gaurav Pande 于2023年6月19日周一 11:22写道： > > > > > > > > > > > Hi Guys, > > > > > > > > > > > > Any help on this thread please? > > > > > > > > > > > > Regards, > > > > > > Gaurav > > > > > > > > > > > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, > > > > wrote: > > > > > > > > > > > > > Hello Guys, > > > > > > > > > > > > > > Iam new in this space, I wanted to know process/steps to > > convert a > > > > > > single > > > > > > > Zk node in standalone presently to a 3 node ZK quorum . > > > > > > > > > > > > > > What steps should I perform to add make my standalone Zk node a > > > > Quorum > > > > > of > > > > > > > 3 zk nodes? > > > > > > > > > > > > > > I am using ZK version 3.5.8 presently. > > > > > > > > > > > > > > Note - I have single Zookeeper(Standalone) and 3 Kafka brokers > at > > > > this > > > > > > > point. > > > > > > > > > > > > > > Regards, > > > > > > > Gaurav > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Convert Standalone zookeeper to A 3 node Quorum
On Mon, Jun 19, 2023 at 8:47 AM Gaurav Pande wrote: > Hi Patrick, > > Thanks for guidance here , based on below I only have presently 1 zk node > so if I provision 2 new VM and install same version of zookeeper on them > should I start them as standalone zookeepers first and then make changes to > there server config file? > > No, you specifically want to add them as part of the ensemble, one at a time in sequence (see 2). > Also what are the valid zk configs that I would need to add on these 2 new > VM's zk server config file? > > You'll need a regular ensemble config, not a standalone. Patrick > Regards, > Gaurav > > On Mon, 19 Jun, 2023, 21:06 Patrick Hunt, wrote: > > > Two ways to do it come to mind, which I've used in the past: > > > > 1) most straightforward is to "clone" the repos for the two new members > of > > the ensemble in their respective configs/datadirs. Just make sure to > update > > the configs appropriately. Including the "myid" for each server. Then > > restart the ensemble and verify. > > > > 2) You can't go from 1->3 servers just by adding two new servers to the > > ensemble as they may form a quorum on the "zero" zxid, rather than the > zxid > > of the existing member. Rather you would need to go from 1->2, with > quorum, > > and then from 2->3 with quorum. This will ensure that the true state of > the > > original quorum is maintained (this is what we implemented for Cloudera > > Manager to ensure proper functioning when increasing quorum size). > > > > Good luck, > > > > Patrick > > > > > > On Mon, Jun 19, 2023 at 12:02 AM Gaurav Pande > > wrote: > > > > > Hi tison, > > > > > > When you say stop services you mean existing standalone Zk service > right? > > > , if that's the case then yes we can. But what's the process ? Also can > > it > > > be done without re-start I didn't know , can you share both > > process/steps? > > > > > > On Mon, 19 Jun, 2023, 10:15 tison, wrote: > > > > > > > Can you stop the services for reconfig, or you need an online > reconfig? > > > > > > > > Best, > > > > tison. > > > > > > > > > > > > Gaurav Pande 于2023年6月19日周一 11:22写道： > > > > > > > > > Hi Guys, > > > > > > > > > > Any help on this thread please? > > > > > > > > > > Regards, > > > > > Gaurav > > > > > > > > > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, > > > wrote: > > > > > > > > > > > Hello Guys, > > > > > > > > > > > > Iam new in this space, I wanted to know process/steps to > convert a > > > > > single > > > > > > Zk node in standalone presently to a 3 node ZK quorum . > > > > > > > > > > > > What steps should I perform to add make my standalone Zk node a > > > Quorum > > > > of > > > > > > 3 zk nodes? > > > > > > > > > > > > I am using ZK version 3.5.8 presently. > > > > > > > > > > > > Note - I have single Zookeeper(Standalone) and 3 Kafka brokers at > > > this > > > > > > point. > > > > > > > > > > > > Regards, > > > > > > Gaurav > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Convert Standalone zookeeper to A 3 node Quorum
Hi Patrick, Thanks for guidance here , based on below I only have presently 1 zk node so if I provision 2 new VM and install same version of zookeeper on them should I start them as standalone zookeepers first and then make changes to there server config file? Also what are the valid zk configs that I would need to add on these 2 new VM's zk server config file? Regards, Gaurav On Mon, 19 Jun, 2023, 21:06 Patrick Hunt, wrote: > Two ways to do it come to mind, which I've used in the past: > > 1) most straightforward is to "clone" the repos for the two new members of > the ensemble in their respective configs/datadirs. Just make sure to update > the configs appropriately. Including the "myid" for each server. Then > restart the ensemble and verify. > > 2) You can't go from 1->3 servers just by adding two new servers to the > ensemble as they may form a quorum on the "zero" zxid, rather than the zxid > of the existing member. Rather you would need to go from 1->2, with quorum, > and then from 2->3 with quorum. This will ensure that the true state of the > original quorum is maintained (this is what we implemented for Cloudera > Manager to ensure proper functioning when increasing quorum size). > > Good luck, > > Patrick > > > On Mon, Jun 19, 2023 at 12:02 AM Gaurav Pande > wrote: > > > Hi tison, > > > > When you say stop services you mean existing standalone Zk service right? > > , if that's the case then yes we can. But what's the process ? Also can > it > > be done without re-start I didn't know , can you share both > process/steps? > > > > On Mon, 19 Jun, 2023, 10:15 tison, wrote: > > > > > Can you stop the services for reconfig, or you need an online reconfig? > > > > > > Best, > > > tison. > > > > > > > > > Gaurav Pande 于2023年6月19日周一 11:22写道： > > > > > > > Hi Guys, > > > > > > > > Any help on this thread please? > > > > > > > > Regards, > > > > Gaurav > > > > > > > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, > > wrote: > > > > > > > > > Hello Guys, > > > > > > > > > > Iam new in this space, I wanted to know process/steps to convert a > > > > single > > > > > Zk node in standalone presently to a 3 node ZK quorum . > > > > > > > > > > What steps should I perform to add make my standalone Zk node a > > Quorum > > > of > > > > > 3 zk nodes? > > > > > > > > > > I am using ZK version 3.5.8 presently. > > > > > > > > > > Note - I have single Zookeeper(Standalone) and 3 Kafka brokers at > > this > > > > > point. > > > > > > > > > > Regards, > > > > > Gaurav > > > > > > > > > > > > > > > > > > > >
Re: Convert Standalone zookeeper to A 3 node Quorum
Two ways to do it come to mind, which I've used in the past: 1) most straightforward is to "clone" the repos for the two new members of the ensemble in their respective configs/datadirs. Just make sure to update the configs appropriately. Including the "myid" for each server. Then restart the ensemble and verify. 2) You can't go from 1->3 servers just by adding two new servers to the ensemble as they may form a quorum on the "zero" zxid, rather than the zxid of the existing member. Rather you would need to go from 1->2, with quorum, and then from 2->3 with quorum. This will ensure that the true state of the original quorum is maintained (this is what we implemented for Cloudera Manager to ensure proper functioning when increasing quorum size). Good luck, Patrick On Mon, Jun 19, 2023 at 12:02 AM Gaurav Pande wrote: > Hi tison, > > When you say stop services you mean existing standalone Zk service right? > , if that's the case then yes we can. But what's the process ? Also can it > be done without re-start I didn't know , can you share both process/steps? > > On Mon, 19 Jun, 2023, 10:15 tison, wrote: > > > Can you stop the services for reconfig, or you need an online reconfig? > > > > Best, > > tison. > > > > > > Gaurav Pande 于2023年6月19日周一 11:22写道： > > > > > Hi Guys, > > > > > > Any help on this thread please? > > > > > > Regards, > > > Gaurav > > > > > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, > wrote: > > > > > > > Hello Guys, > > > > > > > > Iam new in this space, I wanted to know process/steps to convert a > > > single > > > > Zk node in standalone presently to a 3 node ZK quorum . > > > > > > > > What steps should I perform to add make my standalone Zk node a > Quorum > > of > > > > 3 zk nodes? > > > > > > > > I am using ZK version 3.5.8 presently. > > > > > > > > Note - I have single Zookeeper(Standalone) and 3 Kafka brokers at > this > > > > point. > > > > > > > > Regards, > > > > Gaurav > > > > > > > > > > > > > >
RE: Re: Any change from 3.6.3 -> 3.6.4 would cause hostname unresolved issue?
Hi all, We were able to overcome the binding issue by setting quorumListenOnAllIPs=true but from there we are getting a new issue that is preventing leader election completion on first start-up. Getting the log of the current ZooKeeper leader (ID=3) we see the following. (Starting with ** you can see some additional logs added to org.apache.zookeeper.server.quorum.Leader#getDesignatedLeader in order to get more information.) 2023-06-19 12:32:51,990 INFO Have quorum of supporters, sids: [[1, 3],[1, 3]]; starting up and setting last processed zxid: 0x1 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] 2023-06-19 12:32:51,990 INFO ** newQVAcksetPair.getQuorumVerifier().getVotingMembers().get(self.getId()).addr = my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/ 172.17.0.6:2888 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] 2023-06-19 12:32:51,990 INFO ** self.getQuorumAddress() = my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/:2888 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] 2023-06-19 12:32:51,992 INFO ** qs.addr my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/ 172.17.0.6:2888, qs.electionAddr my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc/ 172.17.0.6:3888, qs.clientAddr/127.0.0.1:12181 (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] 2023-06-19 12:32:51,992 DEBUG zookeeper (org.apache.zookeeper.common.PathTrie) [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] 2023-06-19 12:32:51,993 WARN Restarting Leader Election (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] So the leader is ZooKeeper with ID=3 and it was ACKed by the ZooKeeper node ID=1. As you can see we are in the Leader#startZkServer method, and because of the reconfiguration enabled, the designatedLeader is processed. The problem is that the Leader#getDesignatedLeader is not returning “self” as leader but another one (ID=1), because of the difference in the quorum address. >From the above log, it’s not an actual difference in terms of addresses but the self.getQuorumAddress() is returning an one (even if it’s still the same hostname related to ZooKeeper-2 instance). This difference causes the allowedToCommit=false, meanwhile the ZooKeeper-2 is still reported as leader but it’s not able to commit, so prevents any requests and the ZooKeeper ensemble gets stuck. 2023-06-19 12:32:51,996 WARN Suggested leader: 1 (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] 2023-06-19 12:32:51,996 WARN This leader is not the designated leader, it will be initialized with allowedToCommit = false (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=3](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] The overall issue could be related to DNS problems, with DNS records not registered yet during pod initialization (where ZooKeeper is running on Kubernetes). But we don’t understand why it’s not able to recover somehow. Instead of using the quorumListenOnAllIPs=true we also tried a different approach by using the 0.0.0.0 address for the binding, so something like: # Zookeeper nodes configuration server.1=0.0.0.0:2888:3888:participant;127.0.0.1:12181 server.2=my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.default.svc:2888:3888:participant; 127.0.0.1:12181 server.3=my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.default.svc:2888:3888:participant; 127.0.0.1:12181 This way, the self.getQuorumAddress() is not suffering the same problem, it doesn’t return an address but always an actual one. No new leader election is needed and everything works fine. On 2023/06/14 06:19:24 Szalay-Bekő Máté wrote: > Interesting... > > I am not familiar with strimzi.io. > Quickly checking the release notes, I don't see anything suspicious: > https://zookeeper.apache.org/doc/r3.6.4/releasenotes.html > Also, QuorumCnxManager was not changed for 2+ years on branch 3.6. > > Are you use the same java version and zookeeper config for 3.6.3 and 3.6.4? > Can you share the zookeeper config? > > Also: zookeeper 3.6 is deprecated since december 2022. Can you reproduce > the issue on newer ZooKeeper versions? > > best regards, > Máté > > On Tue, Jun 13, 2023 at 10:16 AM Luke Chen wrote: > > > Hi all, > > > > We're running zookeeper under minikube using strimzi < https://strimzi.io/ > > >. > > The zookeeper works well while running with ZK v3.6.3. But when we upgraded > > to v3.6.4, we encountered hostname unresolved issue. I'm wondering if this > > is a regression that some changes between v3.6.3 and v3.6.4 cause this > > issue? > > > > Logs: > > > > 2023-06-12 12:25:38,149 INFO binding to port /127.0.0.1:12181 > >
Re: Convert Standalone zookeeper to A 3 node Quorum
Hi tison, When you say stop services you mean existing standalone Zk service right? , if that's the case then yes we can. But what's the process ? Also can it be done without re-start I didn't know , can you share both process/steps? On Mon, 19 Jun, 2023, 10:15 tison, wrote: > Can you stop the services for reconfig, or you need an online reconfig? > > Best, > tison. > > > Gaurav Pande 于2023年6月19日周一 11:22写道： > > > Hi Guys, > > > > Any help on this thread please? > > > > Regards, > > Gaurav > > > > On Sun, 18 Jun, 2023, 20:14 Gaurav Pande, wrote: > > > > > Hello Guys, > > > > > > Iam new in this space, I wanted to know process/steps to convert a > > single > > > Zk node in standalone presently to a 3 node ZK quorum . > > > > > > What steps should I perform to add make my standalone Zk node a Quorum > of > > > 3 zk nodes? > > > > > > I am using ZK version 3.5.8 presently. > > > > > > Note - I have single Zookeeper(Standalone) and 3 Kafka brokers at this > > > point. > > > > > > Regards, > > > Gaurav > > > > > > > > >
