[
https://issues.apache.org/jira/browse/HDFS-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17746507#comment-17746507
]
ASF GitHub Bot commented on HDFS-17111:
---------------------------------------
simbadzina commented on code in PR #5860:
URL: https://github.com/apache/hadoop/pull/5860#discussion_r1272444972
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##########
@@ -1928,9 +1928,14 @@ public BatchedEntries<OpenFileEntry> listOpenFiles(long
prevId,
@Override
public void msync() throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.READ, true);
- Set<FederationNamespaceInfo> nss = namenodeResolver.getNamespaces();
+ // Only msync to nameservices with observer reads enabled.
+ Set<FederationNamespaceInfo> allNamespaces =
namenodeResolver.getNamespaces();
RemoteMethod method = new RemoteMethod("msync");
- rpcClient.invokeConcurrent(nss, method);
+ Set<FederationNamespaceInfo> namespacesEligibleForObserverReads =
allNamespaces
+ .stream()
+ .filter(ns ->
rpcClient.isNamespaceObserverReadEligible(ns.getNameserviceId()))
+ .collect(Collectors.toSet());
+ rpcClient.invokeConcurrent(namespacesEligibleForObserverReads, method);
Review Comment:
You are right. I added a new test case that was failing with the following
error
```
org.apache.hadoop.ipc.RemoteException(java.io.IOException): No remote
locations available
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1460)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1395)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1366)
```
I've implemented your suggested fix. Thanks for catching this @ayushtkn
> RBF: Optimize msync to only call nameservices that have observer reads
> enabled.
> -------------------------------------------------------------------------------
>
> Key: HDFS-17111
> URL: https://issues.apache.org/jira/browse/HDFS-17111
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Simbarashe Dzinamarira
> Assignee: Simbarashe Dzinamarira
> Priority: Major
> Labels: pull-request-available
>
> Right now when a client MSYNCs to the router, the call is fanned out to all
> nameservices. We only need to proxy the msync to nameservices that have
> observer reads configured.
> We can do this either by adding a new config for the admin to specify which
> nameservices have CRS configured, or we can try to automatically detect these.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
