Bharat Viswanadham created HDDS-3291: ----------------------------------------
Summary: Write operation when both OM followers are shutdown Key: HDDS-3291 URL: https://issues.apache.org/jira/browse/HDDS-3291 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham steps taken : -------------- 1. In OM HA environment, shutdown both OM followers. 2. Start PUT key operation. PUT key operation is hung. Cluster details : https://quasar-vwryte-1.quasar-vwryte.root.hwx.site:7183/cmf/home Snippet of OM log on LEADER: {code:java} 2020-03-24 04:16:46,249 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om2-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:46,249 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om3-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:46,250 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om2: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:46,250 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om3: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:46,750 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om3-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:46,750 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om2-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:46,750 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om3: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:46,750 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om2: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:47,250 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om3-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:47,251 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om2-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:47,251 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om3: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:47,251 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om2: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:47,751 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om2-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:47,751 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om3-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:47,752 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om2: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:47,752 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om3: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:48,252 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om2-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:48,252 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om3-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:48,252 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om2: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:48,252 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om3: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:48,752 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om3-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:48,752 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om2-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:48,753 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om3: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:48,753 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om2: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:49,254 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om2-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:49,254 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om3-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:49,254 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om2: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:49,254 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om3: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:49,754 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om2-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:49,754 WARN org.apache.ratis.grpc.server.GrpcLogAppender: om1@group-9F198C4C3682->om3-AppendLogResponseHandler: Failed appendEntries: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2020-03-24 04:16:49,754 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om2: nextIndex: updateUnconditionally 360 -> 359 2020-03-24 04:16:49,754 INFO org.apache.ratis.server.impl.FollowerInfo: om1@group-9F198C4C3682->om3: nextIndex: updateUnconditionally 360 -> 359 {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org