> ... : n4@group-ABB3109A44C2 replies to PRE_VOTE vote request:
n2<-n4#0:FAIL-t1-last:(t:1, i:16). Peer's state: n4@group-ABB3109A44C2:t1,
leader=n1, voted=null,
raftlog=Memoized:n4@group-ABB3109A44C2-SegmentedRaftLog:OPENED:c16:last(t:1,
i:16), conf=conf: {index: 15, cur=peers:[n1|0.0.0.0:9000, n2|0.0.0.0:9001,
n4|0.0.0.0:9003]|listeners:[], old=null}
> ... : n4@group-ABB3109A44C2: receive requestVote(PRE_VOTE, n2,
group-ABB3109A44C2, 1, (t:1, i:16))
> ... : n4@group-ABB3109A44C2-LISTENER: reject PRE_VOTE from n2: this
server is a listener, who is a non-voting memberAccording to the above log, you actually removed n3. Somehow n4 rejected the voteRequest from n2 and said that it was a non-voting member. It seems like a bug. Could you share all the commands you have run? I could try reproducing it. Thanks. Tsz-Wo On Fri, Jan 16, 2026 at 11:47 AM Tsz Wo Sze <[email protected]> wrote: > Hi Snehasish, > > Thanks for providing the details! > > > 3. After killing n3 and promoting n4 as follower > > ... > > 4. After killing n1 (leader) instance > > As Xinyu mentioned, you probably have changed the voting member size from > 3 to 4. So, killing two servers, n3 and n1, makes it impossible to elect a > new leader since a majority requires 3 voting members. > > I guess you were setConf from [follower: n1,n2,n3; listener: n4] > - to [follower: n1,n2,n3,n4] > > In order to keep voting member size 3, I suggest you to setConf > - to [follower: n1,n2,n4; Listener n3] > (or removing n3, i.e. setConf to [follower: n1,n2, n4]) > > Hope it helps. > Tsz-Wo > > > On Thu, Jan 15, 2026 at 9:44 PM Xinyu Tan <[email protected]> wrote: > >> Hi, >> >> Okay, I understand your question! >> >> Congratulations, you may have found a potential bug. >> >> Next, you can further investigate Ratis's leader election mechanism and >> see why the other two followers refused to vote! >> >> You maybe see some code according to the logs such as[1][2]. >> >> [1] >> https://github.com/apache/ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/RaftServerImpl.java#L1428 >> [2] >> https://github.com/apache/ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderElection.java#L442-L481 >> >> Best >> -------------- >> Xinyu Tan >> >> On 2026/01/16 05:16:04 Snehasish Roy wrote: >> > Hello, >> > >> > Thank you for your prompt response. >> > >> > > You only killed n3 instead of removing it from the cluster, and n1 >> and n2 >> > formed the quorum. >> > I did remove n3 from the cluster before promoting n4 to the follower. >> This >> > was successful because n1 and n2 were still online. >> > >> > This is evident from the below info which shows n1, n2 and n4 are in the >> > cluster. >> > >> > ``` >> > ❯ ./ratis sh group info -peers 0.0.0.0:9000,0.0.0.0:9001,0.0.0.0:9002, >> > 0.0.0.0:9003 -groupid 02511d47-d67c-49a3-9011-abb3109a44c2 >> > [main] WARN org.apache.ratis.metrics.MetricRegistriesLoader - Found >> > multiple MetricRegistries: [class >> > org.apache.ratis.metrics.impl.MetricRegistriesImpl, class >> > org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl]. Using the >> > first: class org.apache.ratis.metrics.impl.MetricRegistriesImpl >> > group id: 02511d47-d67c-49a3-9011-abb3109a44c2 >> > leader info: n1(0.0.0.0:9000) >> > >> > [server { >> > id: "n1" >> > address: "0.0.0.0:9000" >> > startupRole: FOLLOWER >> > } >> > commitIndex: 16 >> > , server { >> > id: "n2" >> > address: "0.0.0.0:9001" >> > startupRole: FOLLOWER >> > } >> > commitIndex: 16 >> > , server { >> > id: "n4" >> > address: "0.0.0.0:9003" >> > startupRole: FOLLOWER >> > } >> > commitIndex: 16 >> > ] >> > applied { >> > term: 1 >> > index: 16 >> > } >> > committed { >> > term: 1 >> > index: 16 >> > } >> > lastEntry { >> > term: 1 >> > index: 16 >> > } >> > ``` >> > >> > After killing n1, the logs from n2 also list the configuration which >> > clearly shows the peers as n1, n2 and n4. >> > Logs also demonstrates that n2 is asking for votes from n1 and n4 - not >> > from n3, indicating the cluster is only a 3 node cluster. >> > >> > ``` >> > INFO [2026-01-15 17:48:03,347] [n2@group-ABB3109A44C2-LeaderElection176 >> ] >> > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection176 PRE_VOTE >> round 0: >> > submit vote requests at term 1 for conf: {index: 15, cur=peers:[n1| >> > 0.0.0.0:9000, n2|0.0.0.0:9001, n4|0.0.0.0:9003]|listeners:[], old=null} >> > ``` >> > >> > Please let me know if I misunderstood something. >> > >> > >> > >> > Regards, >> > Snehasish >> > >> > On Fri, 16 Jan 2026 at 08:49, Xinyu Tan <[email protected]> wrote: >> > >> > > Hi, >> > > >> > > In your scenario, there are two phenomena: >> > > >> > > Phenomenon 1 >> > > You only killed n3 instead of removing it from the cluster, and n1 >> and n2 >> > > formed the quorum. As a result, compared to the last test, you were >> able to >> > > successfully promote n4 from listener to follower, which is as >> expected >> > > because, during the member change, the original quorum of n1, n2, and >> n3, >> > > i.e., n1 and n2, were still online. >> > > >> > > Phenomenon 2 >> > > It is important to note that once you promote n4 to a follower, the >> group >> > > members become n1, n2, n3, and n4. The quorum is now 3 instead of 2, >> and >> > > since n3 has already been killed, killing n1 at this point would >> cause the >> > > consensus group to fail to form a quorum of 3 members, making it >> impossible >> > > to elect a new leader. If you wish to perform this action, you can try >> > > removing the killed n3 from the group first. This way, the consensus >> group >> > > will only consist of n1, n2, and n4, and the quorum will be 2. At this >> > > point, killing n1 should allow the election of a new leader, as the >> quorum >> > > of 2 members is still online. >> > > >> > > Your test scenario involves the most complex part of the consensus >> > > algorithm—member changes. I think you should take a closer look at >> the PhD >> > > thesis of the Raft authors[1], which is more detailed than the ATC2014 >> > > conference version. I believe that after reading it, you will have a >> deeper >> > > understanding of the Raft algorithm! >> > > >> > > Looking forward to your next test. >> > > >> > > [1] https://github.com/ongardie/dissertation/blob/master/stanford.pdf >> > > >> > > Best >> > > ----------------- >> > > Xinyu Tan >> > > >> > > On 2026/01/15 15:20:24 Snehasish Roy wrote: >> > > > Hello, >> > > > >> > > > Based on your inputs, I was able to reproduce the issue >> consistently. >> > > > >> > > > 1. After starting n1, n2 and n3 nodes >> > > > >> > > > ``` >> > > > ./ratis sh group info -peers 0.0.0.0:9000,0.0.0.0:9001,0.0.0.0:9002 >> , >> > > > 0.0.0.0:9003 -groupid 02511d47-d67c-49a3-9011-abb3109a44c2 >> > > > [main] WARN org.apache.ratis.metrics.MetricRegistriesLoader - Found >> > > > multiple MetricRegistries: [class >> > > > org.apache.ratis.metrics.impl.MetricRegistriesImpl, class >> > > > org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl]. >> Using the >> > > > first: class org.apache.ratis.metrics.impl.MetricRegistriesImpl >> > > > group id: 02511d47-d67c-49a3-9011-abb3109a44c2 >> > > > leader info: n1(0.0.0.0:9000) >> > > > >> > > > [server { >> > > > id: "n1" >> > > > address: "0.0.0.0:9000" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 8 >> > > > , server { >> > > > id: "n2" >> > > > address: "0.0.0.0:9001" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 8 >> > > > , server { >> > > > id: "n3" >> > > > address: "0.0.0.0:9002" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 8 >> > > > ] >> > > > applied { >> > > > term: 1 >> > > > index: 8 >> > > > } >> > > > committed { >> > > > term: 1 >> > > > index: 8 >> > > > } >> > > > lastEntry { >> > > > term: 1 >> > > > index: 8 >> > > > } >> > > > ``` >> > > > >> > > > 2. After adding n4 as listener >> > > > >> > > > ``` >> > > > ./ratis sh group info -peers 0.0.0.0:9000,0.0.0.0:9001,0.0.0.0:9002 >> , >> > > > 0.0.0.0:9003 -groupid 02511d47-d67c-49a3-9011-abb3109a44c2 >> > > > [main] WARN org.apache.ratis.metrics.MetricRegistriesLoader - Found >> > > > multiple MetricRegistries: [class >> > > > org.apache.ratis.metrics.impl.MetricRegistriesImpl, class >> > > > org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl]. >> Using the >> > > > first: class org.apache.ratis.metrics.impl.MetricRegistriesImpl >> > > > group id: 02511d47-d67c-49a3-9011-abb3109a44c2 >> > > > leader info: n1(0.0.0.0:9000) >> > > > >> > > > [server { >> > > > id: "n1" >> > > > address: "0.0.0.0:9000" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 12 >> > > > , server { >> > > > id: "n2" >> > > > address: "0.0.0.0:9001" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 12 >> > > > , server { >> > > > id: "n3" >> > > > address: "0.0.0.0:9002" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 12 >> > > > , server { >> > > > id: "n4" >> > > > address: "0.0.0.0:9003" >> > > > startupRole: LISTENER >> > > > } >> > > > commitIndex: 12 >> > > > ] >> > > > applied { >> > > > term: 1 >> > > > index: 12 >> > > > } >> > > > committed { >> > > > term: 1 >> > > > index: 12 >> > > > } >> > > > lastEntry { >> > > > term: 1 >> > > > index: 12 >> > > > } >> > > > ``` >> > > > >> > > > 3. After killing n3 and promoting n4 as follower >> > > > >> > > > ``` >> > > > ❯ ./ratis sh group info -peers 0.0.0.0:9000,0.0.0.0:9001, >> 0.0.0.0:9002, >> > > > 0.0.0.0:9003 -groupid 02511d47-d67c-49a3-9011-abb3109a44c2 >> > > > [main] WARN org.apache.ratis.metrics.MetricRegistriesLoader - Found >> > > > multiple MetricRegistries: [class >> > > > org.apache.ratis.metrics.impl.MetricRegistriesImpl, class >> > > > org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl]. >> Using the >> > > > first: class org.apache.ratis.metrics.impl.MetricRegistriesImpl >> > > > group id: 02511d47-d67c-49a3-9011-abb3109a44c2 >> > > > leader info: n1(0.0.0.0:9000) >> > > > >> > > > [server { >> > > > id: "n1" >> > > > address: "0.0.0.0:9000" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 16 >> > > > , server { >> > > > id: "n2" >> > > > address: "0.0.0.0:9001" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 16 >> > > > , server { >> > > > id: "n4" >> > > > address: "0.0.0.0:9003" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 16 >> > > > ] >> > > > applied { >> > > > term: 1 >> > > > index: 16 >> > > > } >> > > > committed { >> > > > term: 1 >> > > > index: 16 >> > > > } >> > > > lastEntry { >> > > > term: 1 >> > > > index: 16 >> > > > } >> > > > ``` >> > > > >> > > > 4. After killing n1 (leader) instance >> > > > >> > > > ``` >> > > > ❯ ./ratis sh group info -peers 0.0.0.0:9000,0.0.0.0:9001, >> 0.0.0.0:9002, >> > > > 0.0.0.0:9003 -groupid 02511d47-d67c-49a3-9011-abb3109a44c2 >> > > > [main] WARN org.apache.ratis.metrics.MetricRegistriesLoader - Found >> > > > multiple MetricRegistries: [class >> > > > org.apache.ratis.metrics.impl.MetricRegistriesImpl, class >> > > > org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl]. >> Using the >> > > > first: class org.apache.ratis.metrics.impl.MetricRegistriesImpl >> > > > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: >> UNAVAILABLE: >> > > io >> > > > exception >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:368) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:349) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:174) >> > > > at >> > > > >> > > >> org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupList(AdminProtocolServiceGrpc.java:573) >> > > > at >> > > > >> > > >> org.apache.ratis.grpc.client.GrpcClientProtocolClient.groupList(GrpcClientProtocolClient.java:167) >> > > > at >> > > > >> > > >> org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:106) >> > > > at >> > > > >> > > >> org.apache.ratis.client.impl.BlockingImpl.sendRequest(BlockingImpl.java:147) >> > > > at >> > > > >> > > >> org.apache.ratis.client.impl.BlockingImpl.sendRequestWithRetry(BlockingImpl.java:109) >> > > > at >> > > > >> > > >> org.apache.ratis.client.impl.GroupManagementImpl.list(GroupManagementImpl.java:69) >> > > > at >> > > > >> > > >> org.apache.ratis.shell.cli.CliUtils.lambda$getGroupId$1(CliUtils.java:118) >> > > > at >> > > > >> > > >> org.apache.ratis.shell.cli.CliUtils.applyFunctionReturnFirstNonNull(CliUtils.java:72) >> > > > at org.apache.ratis.shell.cli.CliUtils.getGroupId(CliUtils.java:117) >> > > > at >> > > > org.apache.ratis.shell.cli.sh >> > > .command.AbstractRatisCommand.run(AbstractRatisCommand.java:70) >> > > > at >> > > > org.apache.ratis.shell.cli.sh >> > > .group.GroupInfoCommand.run(GroupInfoCommand.java:47) >> > > > at >> org.apache.ratis.shell.cli.AbstractShell.run(AbstractShell.java:104) >> > > > at org.apache.ratis.shell.cli.sh >> .RatisShell.main(RatisShell.java:62) >> > > > Caused by: >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: >> > > > Connection refused: /0.0.0.0:9000 >> > > > Caused by: java.net.ConnectException: Connection refused >> > > > at java.base/sun.nio.ch.Net.pollConnect(Native Method) >> > > > at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) >> > > > at >> > > > java.base/sun.nio.ch >> > > .SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:784) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >> > > > at java.base/java.lang.Thread.run(Thread.java:833) >> > > > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: >> UNAVAILABLE: >> > > io >> > > > exception >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:368) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:349) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:174) >> > > > at >> > > > >> > > >> org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupInfo(AdminProtocolServiceGrpc.java:580) >> > > > at >> > > > >> > > >> org.apache.ratis.grpc.client.GrpcClientProtocolClient.groupInfo(GrpcClientProtocolClient.java:173) >> > > > at >> > > > >> > > >> org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:110) >> > > > at >> > > > >> > > >> org.apache.ratis.client.impl.BlockingImpl.sendRequest(BlockingImpl.java:147) >> > > > at >> > > > >> > > >> org.apache.ratis.client.impl.BlockingImpl.sendRequestWithRetry(BlockingImpl.java:109) >> > > > at org.apache.ratis.client.impl.GroupManagementImpl.info >> > > > (GroupManagementImpl.java:79) >> > > > at >> > > > >> > > >> org.apache.ratis.shell.cli.CliUtils.lambda$getGroupInfo$2(CliUtils.java:146) >> > > > at >> > > > >> > > >> org.apache.ratis.shell.cli.CliUtils.applyFunctionReturnFirstNonNull(CliUtils.java:72) >> > > > at >> org.apache.ratis.shell.cli.CliUtils.getGroupInfo(CliUtils.java:145) >> > > > at >> > > > org.apache.ratis.shell.cli.sh >> > > .command.AbstractRatisCommand.run(AbstractRatisCommand.java:71) >> > > > at >> > > > org.apache.ratis.shell.cli.sh >> > > .group.GroupInfoCommand.run(GroupInfoCommand.java:47) >> > > > at >> org.apache.ratis.shell.cli.AbstractShell.run(AbstractShell.java:104) >> > > > at org.apache.ratis.shell.cli.sh >> .RatisShell.main(RatisShell.java:62) >> > > > Caused by: >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: >> > > > Connection refused: /0.0.0.0:9000 >> > > > Caused by: java.net.ConnectException: Connection refused >> > > > at java.base/sun.nio.ch.Net.pollConnect(Native Method) >> > > > at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) >> > > > at >> > > > java.base/sun.nio.ch >> > > .SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:784) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >> > > > at >> > > > >> > > >> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >> > > > at java.base/java.lang.Thread.run(Thread.java:833) >> > > > group id: 02511d47-d67c-49a3-9011-abb3109a44c2 >> > > > leader info: () >> > > > >> > > > [server { >> > > > id: "n2" >> > > > address: "0.0.0.0:9001" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 16 >> > > > , server { >> > > > id: "n1" >> > > > address: "0.0.0.0:9000" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 16 >> > > > , server { >> > > > id: "n4" >> > > > address: "0.0.0.0:9003" >> > > > startupRole: FOLLOWER >> > > > } >> > > > commitIndex: 16 >> > > > ] >> > > > applied { >> > > > term: 1 >> > > > index: 16 >> > > > } >> > > > committed { >> > > > term: 1 >> > > > index: 16 >> > > > } >> > > > lastEntry { >> > > > term: 1 >> > > > index: 16 >> > > > } >> > > > ``` >> > > > >> > > > Logs from n4 >> > > > ``` >> > > > INFO [2026-01-15 17:48:06,696] [grpc-default-executor-2] >> > > > [RaftServer$Division]: n4@group-ABB3109A44C2 replies to PRE_VOTE >> vote >> > > > request: n2<-n4#0:FAIL-t1-last:(t:1, i:16). Peer's state: >> > > > n4@group-ABB3109A44C2:t1, leader=n1, voted=null, >> > > > raftlog=Memoized:n4@group-ABB3109A44C2-SegmentedRaftLog >> > > :OPENED:c16:last(t:1, >> > > > i:16), conf=conf: {index: 15, cur=peers:[n1|0.0.0.0:9000, n2| >> > > 0.0.0.0:9001, >> > > > n4|0.0.0.0:9003]|listeners:[], old=null} >> > > > INFO [2026-01-15 17:48:06,897] [grpc-default-executor-2] >> > > > [RaftServer$Division]: n4@group-ABB3109A44C2: receive >> > > requestVote(PRE_VOTE, >> > > > n2, group-ABB3109A44C2, 1, (t:1, i:16)) >> > > > INFO [2026-01-15 17:48:06,897] [grpc-default-executor-2] >> [VoteContext]: >> > > > n4@group-ABB3109A44C2-LISTENER: reject PRE_VOTE from n2: this >> server is >> > > a >> > > > listener, who is a non-voting member >> > > > ``` >> > > > >> > > > >> > > > Logs from n2 >> > > > >> > > > ``` >> > > > INFO [2026-01-15 17:48:03,347] >> [n2@group-ABB3109A44C2-LeaderElection176 >> > > ] >> > > > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection176 PRE_VOTE >> > > round 0: >> > > > submit vote requests at term 1 for conf: {index: 15, cur=peers:[n1| >> > > > 0.0.0.0:9000, n2|0.0.0.0:9001, n4|0.0.0.0:9003]|listeners:[], >> old=null} >> > > > INFO [2026-01-15 17:48:03,348] >> [n2@group-ABB3109A44C2-LeaderElection176 >> > > ] >> > > > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection176 got >> exception >> > > > when requesting votes: java.util.concurrent.ExecutionException: >> > > > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: >> UNAVAILABLE: >> > > io >> > > > exception >> > > > INFO [2026-01-15 17:48:03,352] >> [n2@group-ABB3109A44C2-LeaderElection176 >> > > ] >> > > > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection176: PRE_VOTE >> > > > REJECTED received 1 response(s) and 1 exception(s): >> > > > INFO [2026-01-15 17:48:03,352] >> [n2@group-ABB3109A44C2-LeaderElection176 >> > > ] >> > > > [LeaderElection]: Response 0: n2<-n4#0:FAIL-t1-last:(t:1, i:16) >> > > > INFO [2026-01-15 17:48:03,352] >> [n2@group-ABB3109A44C2-LeaderElection176 >> > > ] >> > > > [LeaderElection]: Exception 1: >> java.util.concurrent.ExecutionException: >> > > > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: >> UNAVAILABLE: >> > > io >> > > > exception >> > > > ``` >> > > > >> > > > This indicates that the cluster is in an unstable state. >> > > > I am willing to contribute, could you guide me a bit on this? >> > > > >> > > > >> > > > Regards, >> > > > Snehasish >> > > > >> > > > On Wed, 14 Jan 2026 at 08:44, Snehasish Roy < >> [email protected]> >> > > > wrote: >> > > > >> > > > > Hello, >> > > > > >> > > > > >> > > > > Thank you for your inputs. I will check and update this thread. >> > > > > >> > > > > >> > > > > Regards, >> > > > > Snehasish >> > > > > >> > > > > On Wed, 7 Jan, 2026, 8:52 am Xinyu Tan, <[email protected]> >> wrote: >> > > > > >> > > > >> Hi,Snehasish >> > > > >> >> > > > >> In your scenario, if you kill n3, which is acting as a follower, >> the >> > > > >> cluster will have 3 non-listener and 1 listener, with one >> follower >> > > already >> > > > >> offline. At this point, the majority situation becomes quite >> risky >> > > because >> > > > >> if any non-listener goes down from here, the Raft group will not >> be >> > > able to >> > > > >> form a quorum and elect a new leader. >> > > > >> >> > > > >> Although you have promoted n4 to a listener and removed n3, >> before >> > > this >> > > > >> request completes, the majority of the Raft group is still 2. >> > > Therefore, >> > > > >> after you kill n1, a new leader cannot be elected. In my >> > > understanding, >> > > > >> this phenomenon is not a bug and aligns with the expected >> behavior of >> > > the >> > > > >> algorithm. >> > > > >> >> > > > >> If you want to test how to safely promote a listener to a >> follower, >> > > make >> > > > >> sure that before the promotion request completes (you can confirm >> > > this with >> > > > >> shell commands as suggested by sze), the current leader and >> follower >> > > > >> members maintain the majority online. Otherwise, the promotion >> action >> > > will >> > > > >> not be successful, and this is not a problem with the >> implementation >> > > but a >> > > > >> boundary of the Raft algorithm. >> > > > >> >> > > > >> Feel free to do more testing on this feature of Ratis. If you >> > > encounter >> > > > >> the following issues, it would indicate that there is indeed a >> > > problem with >> > > > >> the implementation, and we welcome discussions and contributions: >> > > > >> * You find that even with the majority of leader and follower >> members >> > > > >> online, you still cannot successfully promote a listener to a >> > > follower. >> > > > >> * In your case, because the majority was not maintained, the >> member >> > > > >> change failed. But after you restart n1 or n3 and re-establish >> the >> > > > >> majority, the Raft group still cannot elect a leader or elects a >> > > leader but >> > > > >> fails to perform member changes. >> > > > >> >> > > > >> We look forward to your testing. >> > > > >> >> > > > >> Best >> > > > >> -------------- >> > > > >> Xinyu Tan >> > > > >> >> > > > >> >> > > > >> On 2025/12/29 10:53:40 Snehasish Roy wrote: >> > > > >> > Hello everyone, >> > > > >> > >> > > > >> > Happy Holidays. This is my first email to this community so >> kindly >> > > > >> excuse >> > > > >> > me for any mistakes. >> > > > >> > >> > > > >> > I initially started a 3 node Ratis Cluster and then added a >> > > listener in >> > > > >> the >> > > > >> > Cluster using the setConfiguration(List.of(n1,n2,n3), >> List.of(n4)) >> > > > >> based on >> > > > >> > the following documentation >> > > > >> > >> > > >> https://jojochuang.github.io/ratis-site/docs/developer-guide/listeners >> > > > >> > >> > > > >> > ``` >> > > > >> > INFO [2025-12-29 15:57:01,887] [n1-server-thread1] >> > > > >> [RaftServer$Division]: >> > > > >> > n1@group-ABB3109A44C2-LeaderStateImpl: startSetConfiguration >> > > > >> > >> SetConfigurationRequest:client-044D31187FB4->n1@group-ABB3109A44C2, >> > > > >> cid=3, >> > > > >> > seq=null, RW, null, SET_UNCONDITIONALLY, servers:[n1| >> 0.0.0.0:9000, >> > > n2| >> > > > >> > 0.0.0.0:9001, n3|0.0.0.0:9002], listeners:[n4|0.0.0.0:9003] >> > > > >> > ``` >> > > > >> > >> > > > >> > Then I killed one of the Ratis follower node (n3) followed by >> > > promoting >> > > > >> the >> > > > >> > listener to the follower using >> setConfiguration(List.of(n1,n2,n4)) >> > > > >> command >> > > > >> > to maintain the cluster size of 3. >> > > > >> > Please note that n3 has been removed from the list of >> followers and >> > > > >> there >> > > > >> > are no more listeners in the cluster and there were no failures >> > > observed >> > > > >> > while issuing the command. >> > > > >> > >> > > > >> > ``` >> > > > >> > INFO [2025-12-29 16:02:54,227] [n1-server-thread2] >> > > > >> [RaftServer$Division]: >> > > > >> > n1@group-ABB3109A44C2-LeaderStateImpl: startSetConfiguration >> > > > >> > >> SetConfigurationRequest:client-2438CA24E2F3->n1@group-ABB3109A44C2, >> > > > >> cid=4, >> > > > >> > seq=null, RW, null, SET_UNCONDITIONALLY, servers:[n1| >> 0.0.0.0:9000, >> > > n2| >> > > > >> > 0.0.0.0:9001, n4|0.0.0.0:9003], listeners:[] >> > > > >> > ``` >> > > > >> > >> > > > >> > Then I killed the leader instance n1. Post which n2 attempted >> to >> > > become >> > > > >> a >> > > > >> > leader and starts asking for votes from n1 and n4. There is no >> > > response >> > > > >> > from n1 as it's not alive and n4 is rejecting the pre_vote >> request >> > > from >> > > > >> n2 >> > > > >> > because it still thinks it's a listener. >> > > > >> > >> > > > >> > Logs from n2 >> > > > >> > ``` >> > > > >> > INFO [2025-12-29 16:04:10,051] >> > > [n2@group-ABB3109A44C2-LeaderElection30 >> > > > >> ] >> > > > >> > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection30 >> PRE_VOTE >> > > > >> round 0: >> > > > >> > submit vote requests at term 1 for conf: {index: 15, >> cur=peers:[n1| >> > > > >> > 0.0.0.0:9000, n2|0.0.0.0:9001, n4|0.0.0.0:9003]|listeners:[], >> > > old=null} >> > > > >> > INFO [2025-12-29 16:04:10,052] >> > > [n2@group-ABB3109A44C2-LeaderElection30 >> > > > >> ] >> > > > >> > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection30 got >> > > exception >> > > > >> when >> > > > >> > requesting votes: java.util.concurrent.ExecutionException: >> > > > >> > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: >> > > > >> UNAVAILABLE: io >> > > > >> > exception >> > > > >> > INFO [2025-12-29 16:04:10,054] >> > > [n2@group-ABB3109A44C2-LeaderElection30 >> > > > >> ] >> > > > >> > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection30: >> PRE_VOTE >> > > > >> REJECTED >> > > > >> > received 1 response(s) and 1 exception(s): >> > > > >> > INFO [2025-12-29 16:04:10,054] >> > > [n2@group-ABB3109A44C2-LeaderElection30 >> > > > >> ] >> > > > >> > [LeaderElection]: Response 0: n2<-n4#0:FAIL-t1-last:(t:1, >> i:16) >> > > > >> > INFO [2025-12-29 16:04:10,054] >> > > [n2@group-ABB3109A44C2-LeaderElection30 >> > > > >> ] >> > > > >> > [LeaderElection]: Exception 1: >> > > > >> java.util.concurrent.ExecutionException: >> > > > >> > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: >> > > > >> UNAVAILABLE: io >> > > > >> > exception >> > > > >> > ``` >> > > > >> > >> > > > >> > >> > > > >> > Due to lack of leader, the cluster is no more stable. >> > > > >> > >> > > > >> > Logs from n4 >> > > > >> > ``` >> > > > >> > INFO [2025-12-29 16:05:03,405] [grpc-default-executor-2] >> > > > >> > [RaftServer$Division]: n4@group-ABB3109A44C2: receive >> > > > >> requestVote(PRE_VOTE, >> > > > >> > n2, group-ABB3109A44C2, 1, (t:1, i:16)) >> > > > >> > INFO [2025-12-29 16:05:03,405] [grpc-default-executor-2] >> > > [VoteContext]: >> > > > >> > n4@group-ABB3109A44C2-LISTENER: reject PRE_VOTE from n2: this >> > > server >> > > > >> is a >> > > > >> > listener, who is a non-voting member >> > > > >> > INFO [2025-12-29 16:05:03,405] [grpc-default-executor-2] >> > > > >> > [RaftServer$Division]: n4@group-ABB3109A44C2 replies to >> PRE_VOTE >> > > vote >> > > > >> > request: n2<-n4#0:FAIL-t1-last:(t:1, i:16). Peer's state: >> > > > >> > n4@group-ABB3109A44C2:t1, leader=n1, voted=null, >> > > > >> > raftlog=Memoized:n4@group-ABB3109A44C2-SegmentedRaftLog >> > > > >> :OPENED:c16:last(t:1, >> > > > >> > i:16), conf=conf: {index: 15, cur=peers:[n1|0.0.0.0:9000, n2| >> > > > >> 0.0.0.0:9001, >> > > > >> > n4|0.0.0.0:9003]|listeners:[], old=null} >> > > > >> > ``` >> > > > >> > >> > > > >> > So my question is how to correctly promote a listener to a >> follower? >> > > > >> Did I >> > > > >> > miss some step? Or is there a bug in the code? If it's the >> latter, I >> > > > >> would >> > > > >> > be happy to contribute. Please let me know if you need any more >> > > > >> debugging >> > > > >> > information. >> > > > >> > >> > > > >> > Thank you again for looking into this issue. >> > > > >> > >> > > > >> > >> > > > >> > Regards, >> > > > >> > Snehasish >> > > > >> > >> > > > >> >> > > > > >> > > > >> > > >> > >> >
