Hi Ben ,
What you said is correct.
Here is the situation in a little more detail:
- My cluster is made of many nodes some which are simple nodes (Single VM) lets
calls them S1, S2, S3 etc and some are duplex nodes D1, D2 , etc.( a pair of
two VMs acting as active-standby)
- Let's say the pair of two VMs in duplex system are named A and B. So we have
D1-A: D1-B, D2-A :D2-B etc.
- I start all nodes at the same time:
- Both D1-A , D1-B try to create an ephemeral node D1
- S1,S2,S3 create an ephemeral node S1,S2,S3 etc.
- D2-A, D2-B try create an ephemeral node D2
- Now let's say following nodes are created (notes all VMs were started at the
same time,, so I cannot control in which order nodes get created)
- D1_000000000
- S1_000000001
- S2_000000002
-D1_000000003
-D2_000000004
-D2_000000005
- Note that there are two nodes for D1 and D2 : that is because each of the
servers D1-A, D1-B, D2-A and D2-B succeed in creating the nodes.
- Clearly D1-A (which created node D1_00000000) becomes the master ... all is
good till this time.
- Now during normal course of operations.... something bad happens on the node
D1-A and it goes away. D1-B being the standby takes over the responsibilities
of D1-A
- However, according to the order above S1 will become the next leader of the
cluster.
- But what I would like to happen is that D1-B becomes the leader of the lot.
Is this possible in any way ?
Hope I am making my self clear.
Thanks
-Parag
On 2/13/21, 12:16 AM, "Benjamin Reed" <[email protected]> wrote:
the simple leader election should work for you: both try to create an
ephemeral node: /leader. the one that succeeds is the leader. sets a
watch on /leader and will try to create it if it goes away. if the
standby succeeds, it will become the leader and will remain so.
(i realize i'm repeating what flavio said :) i though perhaps
different words would help :) is there some functionality that this
solution is missing? it is one of the most popular ways to do leader
election with zookeeper.
ben
On Sat, Jan 30, 2021 at 1:35 AM Dipti Mulay <[email protected]> wrote:
>
> Hi Flavio,
>
> I am using the Zookeepers to do a leader election. The leader election
> needs to be done between entities which are actually a combination of
> active/stanb-by machines. The standby is a hot standby ready to take over
> when the active fails. Now when such a failure occurs I need to ensure
that
> if the active system was the leader then the standby system remains as
> leader when it takes over and becomes active. Now I can have both active
> and standby systems create their own ephemeral node. But that will not
> guarantee that every time a failure happens on the active system the
> standby which is going to become active will assume the role of leader;
> that is because other ephemeral nodes might exist already.
>
> I am hoping that I explained it properly.
>
> Thanks for bearing with me.
> -Parag
>
> On Fri, 29 Jan 2021 at 20:25, Flavio Junqueira <[email protected]> wrote:
>
> > If the session expires, then you cannot reconnect to that session and
the
> > associated ephemerals should be deleted.
> >
> > I'm not entirely clear why you want to preserve the ephemeral znode. The
> > session expiring and the ephemeral being deleted is an indication that
the
> > active leadership has been released, which brings to me to the question
of
> > how the standby is checking whether the active is still up. If it is
not by
> > watching the ephemeral znode, then it must be by other means.
> >
> > If you need multiple clients to know which client is active, then you
> > might consider writing an id to a persistent znode. That alone won't be
> > sufficient, though.
> >
> > -Flavio
> >
> > > On 29 Jan 2021, at 15:03, Dipti Mulay <[email protected]> wrote:
> > >
> > > In his mail Ben said "when the session expires, the ephemeral nodes
will
> > > get deleted. if your standby system reconnects to zookeeper with the
> > > clientid, the session will not expire and the ephemeral nodes will
stay
> > > active."
> > > Are you suggesting that I can re-connect on a session that has already
> > > expired..? Won't the eph node get deleted when the session expires?
> > >
> > > On Fri, 29 Jan 2021 at 18:33, Dipti Mulay <[email protected]> wrote:
> > >
> > >> Hi Ben, Flavio,
> > >>
> > >> Thanks for your replies.
> > >>
> > >> I have tried coding it in a way that when standby takes over from the
> > >> active system only then it will establish a session with ZK server
using
> > >> the same client id. But there is a small window of time when both
> > systems
> > >> have an active session.
> > >> What I want to happen is that when standby takes over and active goes
> > down
> > >> the ephemeral node should remain intact. The reason being that if the
> > >> active server was the "leader" then after the standby takes over I
would
> > >> like the standby to remain as leader.
> > >> I have tried it many ways but as soon as the standby takes over and
> > socket
> > >> from the old active closes the ephemeral node is deleted and triggers
> > >> leader election again.
> > >>
> > >> Hope this helps.
> > >>
> > >> Thanks
> > >> -Parag
> > >>
> > >>
> > >> On Fri, 29 Jan 2021 at 13:26, Flavio Junqueira <[email protected]>
wrote:
> > >>
> > >>> Yeah, what Ben said... What you might want to do is something along
the
> > >>> lines of:
> > >>>
> > >>> 1- Have a client create the ephemeral znode, let's call it say
> > "/active"
> > >>> 2- The other client acting as a standby sets a watch on "/active"
> > >>> 3- If the active node crashes, then "/active" is removed and the
> > standby
> > >>> is notified and can become active by creating the "/active" znode
> > >>>
> > >>> Would that solve your problem?
> > >>>
> > >>> -Flavio
> > >>>
> > >>>> On 29 Jan 2021, at 07:44, Benjamin Reed <[email protected]> wrote:
> > >>>>
> > >>>> can you explain a bit more what you mean by "The zookeeper-client
> > >>>> process on standby which uses the same clientid to initialize a
> > >>>> session with zookeeper." ? if i understand correctly, you are
going to
> > >>>> have two clients using the same session which cannot happen. do you
> > >>>> only reuse the clientid when the takeover occurs (the other node
stops
> > >>>> using the session)?
> > >>>>
> > >>>> when the session expires, the ephemeral nodes will get deleted. if
> > >>>> your standby system reconnects to zookeeper with the clientid, the
> > >>>> session will not expire and the ephemeral nodes will stay active.
> > >>>>
> > >>>> ben
> > >>>>
> > >>>> On Wed, Jan 27, 2021 at 3:20 AM Dipti Mulay <[email protected]>
> > wrote:
> > >>>>>
> > >>>>> Hi Flavio,
> > >>>>>
> > >>>>> Thanks for your inputs let me say what I am trying to do:
> > >>>>>
> > >>>>> I have an active- standby system, each of these system is running
on
> > >>> its own VM.
> > >>>>>
> > >>>>> The zookpeeper client process which communicates with ZK is
running
> > >>> even on the stanby system. The active system creates a session with
> > ZK and
> > >>> also creates a ephemeral node, I use the zoo_client_id() call to get
> > the
> > >>> clientId_t information.
> > >>>>> The active system then passes on this ClientID_t to standby
system(
> > >>> this is done by a memory shadow mechanism we have). The
> > zookeeper-client
> > >>> process on standby which uses the same clientid to initialize a
session
> > >>> with zookeeper.
> > >>>>>
> > >>>>> Now after some time I turn of the active VM. On the stanby I see
logs
> > >>> which tell me a socket error happened and I get a session expired
event
> > >>> upon which I close the zookeeper session. Does the session expire
even
> > >>> also mean that my ephemeral is now gone? The standard error handling
> > in my
> > >>> code is that when a session expired event is received I call
> > >>> zookeeper_close() which will surely remove the ephemeral node.
> > >>>>> Is there a way to preserve the ephemeral node and re-connect the
> > >>> session ?
> > >>>>>
> > >>>>> Thanks
> > >>>>> -Parag
> > >>>>> P.S: I hope I have clearly explained the problem
> > >>>>>
> > >>>>> On 1/26/21, 8:17 PM, "Flavio Junqueira" <[email protected]> wrote:
> > >>>>>
> > >>>>> Hi Parag,
> > >>>>>
> > >>>>> Typically, the application knows the path of the znode that it
> > >>> wants to check. Knowing the path, you can, for example, check the
znode
> > >>> metadata, which includes the session id indicating the owner in the
> > case of
> > >>> an ephemeral znode.
> > >>>>>
> > >>>>> -Flavio
> > >>>>>
> > >>>>>> On 26 Jan 2021, at 03:07, Dipti Mulay <[email protected]> wrote:
> > >>>>>>
> > >>>>>> Hi All,
> > >>>>>>
> > >>>>>> I had a question regarding ephemeral nodes. Given a particular
> > >>> client_id can zookeeper return me the ephemeral node corresponding
to
> > the
> > >>> provided client_id?
> > >>>>>> I am using the C Client Library and I could not find a way ?
> > >>>>>>
> > >>>>>> Any suggestions ?
> > >>>>>>
> > >>>>>> Thanks
> > >>>>>> -Parag
> > >>>>>
> > >>>
> > >>>
> >
> >
