smengcl commented on code in PR #11763:
URL: https://github.com/apache/ozone/pull/11763#discussion_r4476241902
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/ha/OMFailoverProxyProviderBase.java:
##########
@@ -226,13 +226,10 @@ public RetryAction shouldRetry(Exception exception, int
retries,
if (notLeaderException != null) {
// Prepare the next OM to be tried. This will help with calculation
// of the wait times needed get creating the retryAction.
- String suggestedLeaderAddress =
- notLeaderException.getSuggestedLeaderAddress();
String suggestedNodeId =
notLeaderException.getSuggestedLeaderNodeId();
- if (suggestedLeaderAddress != null &&
- suggestedNodeId != null &&
- omProxies.contains(suggestedNodeId, suggestedLeaderAddress)) {
+ if (suggestedNodeId != null && omProxies.get(suggestedNodeId) !=
null
+ && !isAttempted(suggestedNodeId)) {
Review Comment:
`attemptedOMs` is only cleared in `getWaitTime()`, not after a request
succeeds, so it can keep an OM from an earlier failover and this check then
skips a correct suggestion.
Example with om1, om2, om3: om1 is down, a request fails over to om2 by
round robin (`attemptedOMs` = {om1}), and om2 serves it. Later om1 comes back
as leader. om2 suggests om1, but `isAttempted("om1")` is true, so the client
tries om3 first and reaches om1 only after a full round plus a 2s
`waitBetweenRetries` sleep. A small test against `getRetryPolicy` at this head
confirms it.
Could the set be cleared on success, or tracked per request? That keeps the
skip for a down suggested leader.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/grpc/OMGrpcError.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.protocolPB.grpc;
+
+import io.grpc.Metadata;
+import io.grpc.Status;
+import io.grpc.StatusRuntimeException;
+import io.grpc.protobuf.ProtoUtils;
+import java.io.IOException;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMLeaderNotReadyException;
+import org.apache.hadoop.ozone.om.exceptions.OMNotLeaderException;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMGrpcErrorProto;
+import org.apache.ratis.protocol.exceptions.ReadException;
+import org.apache.ratis.protocol.exceptions.ReadIndexException;
+
+/**
+ * The Java form of {@link OMGrpcErrorProto}, the error that the OM gRPC
server sends in the gRPC trailers.
+ */
+public final class OMGrpcError {
+ private static final Metadata.Key<OMGrpcErrorProto> ERROR_KEY =
Metadata.Key.of("om-grpc-error-bin",
+ ProtoUtils.metadataMarshaller(OMGrpcErrorProto.getDefaultInstance()));
+
+ private final Kind kind;
+ private final String message;
+ /** The OM node ID of the suggested leader, for {@link Kind#NOT_LEADER}. */
+ private final String suggestedLeaderNodeId;
+
+ /** A kind for each error in {@link OMGrpcErrorProto}, and {@link #INTERNAL}
for any other error. */
+ public enum Kind {
+ NOT_LEADER,
+ LEADER_NOT_READY,
+ READ_INDEX_FAILED,
+ READ_FAILED,
+ INTERNAL
+ }
+
+ private OMGrpcError(Kind kind, String message, String suggestedLeaderNodeId)
{
+ this.kind = kind;
+ this.message = message;
+ this.suggestedLeaderNodeId = suggestedLeaderNodeId;
+ }
+
+ /** @return the error for the given exception thrown by OM. */
+ public static OMGrpcError fromException(Throwable omException) {
+ if (omException instanceof OMNotLeaderException) {
+ return new OMGrpcError(Kind.NOT_LEADER, omException.getMessage(),
+ ((OMNotLeaderException) omException).getSuggestedLeaderNodeId());
+ } else if (omException instanceof OMLeaderNotReadyException) {
+ return new OMGrpcError(Kind.LEADER_NOT_READY, omException.getMessage(),
null);
+ } else if (omException instanceof ReadIndexException) {
+ return new OMGrpcError(Kind.READ_INDEX_FAILED, omException.getMessage(),
null);
+ } else if (omException instanceof ReadException) {
+ return new OMGrpcError(Kind.READ_FAILED, omException.getMessage(), null);
+ }
+ return new OMGrpcError(Kind.INTERNAL, omException.toString(), null);
+ }
+
+ public static OMGrpcError getFromProtobuf(OMGrpcErrorProto proto) {
+ switch (proto.getErrorCase()) {
+ case NOTLEADER:
+ final OMGrpcErrorProto.NotLeader notLeader = proto.getNotLeader();
+ return new OMGrpcError(Kind.NOT_LEADER, notLeader.hasMessage() ?
notLeader.getMessage() : null,
+ notLeader.hasSuggestedLeaderNodeId() ?
notLeader.getSuggestedLeaderNodeId() : null);
+ case LEADERNOTREADY:
+ final OMGrpcErrorProto.LeaderNotReady leaderNotReady =
proto.getLeaderNotReady();
+ return new OMGrpcError(Kind.LEADER_NOT_READY,
+ leaderNotReady.hasMessage() ? leaderNotReady.getMessage() : null,
null);
+ case READINDEXFAILED:
+ final OMGrpcErrorProto.ReadIndexFailed readIndexFailed =
proto.getReadIndexFailed();
+ return new OMGrpcError(Kind.READ_INDEX_FAILED,
+ readIndexFailed.hasMessage() ? readIndexFailed.getMessage() : null,
null);
+ case READFAILED:
+ final OMGrpcErrorProto.ReadFailed readFailed = proto.getReadFailed();
+ return new OMGrpcError(Kind.READ_FAILED, readFailed.hasMessage() ?
readFailed.getMessage() : null, null);
+ default:
+ return new OMGrpcError(Kind.INTERNAL, null, null);
+ }
+ }
+
+ /**
+ * @return the error that the gRPC error carries, or null if the gRPC error
does not come from OM.
+ */
+ public static OMGrpcError fromStatusRuntimeException(StatusRuntimeException
e) {
+ final Metadata trailers = e.getTrailers();
+ final OMGrpcErrorProto proto = trailers == null ? null :
trailers.get(ERROR_KEY);
+ if (proto == null) {
+ return null;
Review Comment:
With a new S3 Gateway and an older OM (no trailer), this returns null, so
`GrpcOMFollowerReadFailoverProxyProvider#submitRequest` no longer sees
`OMNotLeaderException` and never calls `disableFollowerRead()`. Master does
disable it on the first not-leader error from a follower. With follower read
enabled, reads keep going to followers first, and with `LOCAL_LEASE` every read
tries each follower before falling back to the leader.
A small fallback on the description would fix it (the same could cover
`OMLeaderNotReadyException`):
```diff
---
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/grpc/OMGrpcError.java
+++
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/grpc/OMGrpcError.java
@@ -100,7 +100,11 @@
final Metadata trailers = e.getTrailers();
final OMGrpcErrorProto proto = trailers == null ? null :
trailers.get(ERROR_KEY);
if (proto == null) {
- return null;
+ // OM of older versions only sends "class: message" in the
description.
+ final String description = e.getStatus().getDescription();
+ final String prefix = OMNotLeaderException.class.getName() + ": ";
+ return description != null && description.startsWith(prefix)
+ ? new OMGrpcError(Kind.NOT_LEADER,
description.substring(prefix.length()), null) : null;
}
final OMGrpcError error = getFromProtobuf(proto);
return error.kind == Kind.INTERNAL ? new OMGrpcError(Kind.INTERNAL,
e.getStatus().getDescription(), null) : error;
```
Or mention it in the compatibility note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
