szetszwo commented on code in PR #10486:
URL: https://github.com/apache/ozone/pull/10486#discussion_r3430323286
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java:
##########
@@ -604,6 +604,19 @@ public final class OzoneConfigKeys {
public static final boolean OZONE_JVM_NETWORK_ADDRESS_CACHE_ENABLED_DEFAULT =
true;
+ /**
+ * When true, RPC clients (DN heartbeat, OM client, SCM client) re-resolve
+ * cached hostnames on connection failure and rebuild the proxy if the
+ * resolved IP has changed. Set to true in environments where server pod
+ * IPs may change while DNS names remain stable, such as Kubernetes.
+ * Default false preserves pre-fix behavior. Mirrors the design intent of
+ * HADOOP-17068 / HDFS-14118.
+ */
+ public static final String OZONE_CLIENT_FAILOVER_RESOLVE_NEEDED_KEY =
+ "ozone.client.failover.resolve-needed";
Review Comment:
Move it next to "ozone.client.failover.max.attempts".
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/ConnectionFailureUtils.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils;
+
+import java.io.EOFException;
+import java.net.ConnectException;
+import java.net.NoRouteToHostException;
+import java.net.SocketException;
+import java.net.SocketTimeoutException;
+import java.net.UnknownHostException;
+
+/**
+ * Shared classifier for exceptions where the cached peer IP is no longer
+ * reachable and DNS re-resolution is the only plausible recovery path.
+ * <p>
+ * Used by both {@code SCMFailoverProxyProviderBase} and
+ * {@code OMFailoverProxyProviderBase} to gate the DNS-refresh-on-failure
+ * code path so that application-level errors (NotLeader, AccessControl,
+ * OMException, RetryAction) do not trigger spurious DNS lookups.
+ * <p>
+ * The classifier must match the failure shapes seen in production
+ * Kubernetes deployments where the peer pod has been rescheduled to a
+ * new IP under a stable hostname:
+ * <ul>
+ * <li>{@link ConnectException} -- the TCP SYN was refused. Seen on
+ * OpenStack / fast-RST environments. </li>
+ * <li>{@link SocketTimeoutException} (and its IPC subclass
+ * {@code ConnectTimeoutException}) -- the SYN was dropped silently.
+ * This is the dominant failure shape on AWS EC2 / EKS where the
+ * network silently drops packets to a defunct pod IP. The PR that
+ * introduced this helper (HDDS-15514) is sold on this case; it
+ * must be in the filter. </li>
+ * <li>{@link NoRouteToHostException} -- routing table no longer
+ * reaches the cached IP. </li>
+ * <li>{@link UnknownHostException} -- the hostname itself failed to
+ * resolve at the time the IPC layer reconstructed the address. </li>
+ * <li>{@link EOFException} -- a load balancer or iptables RST closed
+ * the half-open connection cleanly. Common in Kubernetes when an
+ * IP is reassigned to an unrelated pod that rejects the RPC
+ * handshake. </li>
+ * <li>{@link SocketException} (e.g. "Connection reset") -- the peer
+ * sent RST mid-stream. </li>
+ * </ul>
+ * The walk is bounded to {@value #MAX_CAUSE_DEPTH} levels to defend
+ * against cause chains that have been constructed (in violation of
+ * {@code Throwable.initCause}'s contract) into a cycle of length > 1.
+ */
+public final class ConnectionFailureUtils {
+
+ /**
+ * Maximum depth of the {@code Throwable.getCause()} chain we walk
+ * before giving up. Matches Hadoop's own walkers in
+ * {@code RemoteException} handling.
+ */
+ static final int MAX_CAUSE_DEPTH = 16;
+
+ private ConnectionFailureUtils() {
+ }
+
+ /**
+ * Returns true when any link in {@code t}'s cause chain (up to
+ * {@link #MAX_CAUSE_DEPTH} levels) is one of the connection-class
+ * exceptions documented on this class.
+ *
+ * @param t the throwable to classify. {@code null} returns false.
+ */
+ public static boolean isConnectionFailure(Throwable t) {
+ Throwable cause = t;
+ for (int depth = 0; cause != null && depth < MAX_CAUSE_DEPTH; depth++) {
+ if (cause instanceof ConnectException
+ || cause instanceof SocketTimeoutException
+ || cause instanceof NoRouteToHostException
Review Comment:
ConnectException and NoRouteToHostException are subclasses of
SocketException. Let remove them and add a comment.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/ha/OMFailoverProxyProviderBase.java:
##########
@@ -477,4 +509,63 @@ public static ReadException getReadException(Exception
exception) {
protected ConfigurationSource getConf() {
return conf;
}
+
+ /**
+ * Asks the current proxy's {@link OMProxyInfo} to re-resolve its
+ * configured hostname. If DNS now returns a different IP, the
+ * OMProxyInfo replaces its cached address and discards the cached
+ * proxy so the next dial happens against the new IP.
+ * <p>
+ * Calls {@link #onAddressRefreshed(String)} when a swap occurs so
+ * subclasses (specifically {@code HadoopRpcOMFailoverProxyProvider})
+ * can refresh derived state such as the aggregated delegation-token
+ * service identifier.
+ * <p>
+ * The DNS lookup performed inside {@link
OMProxyInfo#refreshAddressIfChanged}
+ * is run OUTSIDE this provider's monitor. {@link OMProxyInfo} maintains
+ * its own monitor for the swap commit; if we held the provider monitor
+ * across the resolve, a slow / dead resolver would freeze every
+ * concurrent caller of synchronized provider methods (e.g.
+ * {@link #performFailover}, {@link #selectNextOmProxy}). The provider
+ * monitor is only re-acquired briefly to invoke the refresh hook.
+ *
+ * @return true if a swap actually happened.
+ */
+ @VisibleForTesting
+ boolean maybeRefreshCurrentOmAddress() {
+ final String nodeId;
+ final OMProxyInfo<T> info;
+ synchronized (this) {
+ nodeId = getCurrentProxyOMNodeId();
+ if (nodeId == null) {
+ return false;
+ }
+ info = omProxies.get(nodeId);
+ if (info == null) {
+ return false;
+ }
+ }
Review Comment:
- nodeId is never null
- info is never null
- getCurrentProxyOMNodeId() is already synchronized
- the omProxies map is unmodifiable
So, I suggest the change below:
```java
final String nodeId = Objects.requireNonNull(getCurrentProxyOMNodeId(),
"Current proxy node id is null");
final OMProxyInfo<T> info =
Objects.requireNonNull(omProxies.get(nodeId), "Current proxy info is null");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
