[
https://issues.apache.org/jira/browse/HDDS-16028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-16028:
-------------------------------
Summary: SCM follower suggests itself as leader can cause OM to be stuck
(was: SCM follower suggests itself as leader causes OM stuck)
> SCM follower suggests itself as leader can cause OM to be stuck
> ---------------------------------------------------------------
>
> Key: HDDS-16028
> URL: https://issues.apache.org/jira/browse/HDDS-16028
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ivan Andika
> Assignee: ChenXi
> Priority: Critical
>
> We have another major incident partly because of SCM follower kept suggesting
> itself as a leader
> {code:java}
> scm15:9894:FOLLOWER:2059f536-5846-4573-b81d-274dc495c727:<redacted>
> scm16:9894:LEADER:d0b9f1ed-7b61-4ac1-89d6-db3bc6fcb9df:<redacted>
> scm17:9894:FOLLOWER:23ef2ecf-888c-4a83-b318-35fc5714ea8b:<redacted>{code}
> {code:java}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException):
> Server:23ef2ecf-888c-4a83-b318-35fc5714ea8b is not the leader. Suggested
> leader is Server:scm17:9863.
> at
> org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:107)
> at
> org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:250)
> at
> org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:117)
> at
> org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:16819)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:495)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:615)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:593)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1153)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1336)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1229)
> at
> java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2031)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3622)
> , while invoking $Proxy34.send over nodeId=scm17,nodeAddress=scm17:9863.
> Trying to failover after sleeping for 2000ms. {code}
> Ratis RaftServerImpl#generateNotLeaderException logic should not have allowed
> it, see below
> {code:java}
> NotLeaderException generateNotLeaderException() {
> if (!lifeCycle.getCurrentState().isRunning()) {
> return new NotLeaderException(getMemberId(), null, null);
> }
> RaftPeerId leaderId = state.getLeaderId();
> if (leaderId == null || leaderId.equals(getId())) {
> // No idea about who is the current leader. Or the peer is the current
> // leader, but it is about to step down. set the suggested leader as null.
> leaderId = null;
> }
> final RaftConfigurationImpl conf = getRaftConf();
> Collection<RaftPeer> peers = conf.getAllPeers();
> return new NotLeaderException(getMemberId(), conf.getPeer(leaderId), peers);
> } {code}
> But because Ozone (annoyingly) reinvented a (buggy) wheel of
> NotLeaderException generation, a non-leader SCM NotLeaderException with
> itself as the suggested leader
> {code:java}
> public NotLeaderException triggerNotLeaderException() {
> ByteString leaderId =
> division.getInfo().getRoleInfoProto().getFollowerInfo().getLeaderInfo()
> .getId().getId();
> RaftPeer suggestedLeader = leaderId.isEmpty() ?
> null :
> division.getRaftConf().getPeer(RaftPeerId.valueOf(leaderId));
> return new NotLeaderException(division.getMemberId(),
> suggestedLeader,
> division.getGroup().getPeers());
> } {code}
> We should use the correct Ratis generatedNotLeaderException.
> This also apply to OM OzoneManagerRatisServer#newOMNotLeaderException, but
> thankfully OM suggested leadership mechanism is broken so it will not be
> triggered (HDDS-14769).
> I will revisit all the Ozone Ratis usage and check if there are some similar
> issues. Most Ozone usage of Ratis need to be moved to Ratis. We will create
> new Ratis API to access it if necessary.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
