[ https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17636802#comment-17636802 ]
ASF GitHub Bot commented on HDFS-16845: --------------------------------------- mkuchenbecker commented on code in PR #5142: URL: https://github.com/apache/hadoop/pull/5142#discussion_r1028372645 ########## hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestObserverWithRouter.java: ########## @@ -439,4 +440,60 @@ public void testRouterMsync() throws Exception { assertEquals("Four calls should be sent to active", 4, rpcCountForActive); } + + @Test + public void testSingleRead() throws Exception { + List<? extends FederationNamenodeContext> namenodes = routerContext + .getRouter().getNamenodeResolver() + .getNamenodesForNameserviceId(cluster.getNameservices().get(0), true); + assertEquals("First namenode should be observer", namenodes.get(0).getState(), + FederationNamenodeServiceState.OBSERVER); + Path path = new Path("/"); + + long rpcCountForActive; + long rpcCountForObserver; + + // Send read request + fileSystem.listFiles(path, false); + fileSystem.close(); + + rpcCountForActive = routerContext.getRouter().getRpcServer() + .getRPCMetrics().getActiveProxyOps(); + // getListingCall sent to active. + assertEquals("Only one call should be sent to active", 1, rpcCountForActive); + + rpcCountForObserver = routerContext.getRouter().getRpcServer() + .getRPCMetrics().getObserverProxyOps(); + // getList call should be sent to observer + assertEquals("No calls should be sent to observer", 0, rpcCountForObserver); + } + + @Test + public void testSingleReadUsingObserverReadProxyProvider() throws Exception { + fileSystem.close(); + fileSystem = routerContext.getFileSystemWithObserverReadProxyProvider(); Review Comment: This seems wrong to special-case in this way. Either manage it during setup or set it for every function, but I'd advise against mixing the two. ########## hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestObserverWithRouter.java: ########## @@ -122,7 +123,9 @@ public void startUpCluster(int numberOfObserver, Configuration confOverrides) th cluster.waitActiveNamespaces(); routerContext = cluster.getRandomRouter(); - fileSystem = routerContext.getFileSystemWithObserverReadsEnabled(); + Configuration confToEnableObserverRead = new Configuration(); + confToEnableObserverRead.setBoolean(HdfsClientConfigKeys.DFS_RBF_OBSERVER_READ_ENABLE, true); + fileSystem = routerContext.getFileSystem(confToEnableObserverRead); Review Comment: We are losing coverage on `getFileSystemWithObserverReadsEnabled` with this change; we should likely be testing both as they are both valid use-cases whether you want to msync or not. ########## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java: ########## @@ -349,6 +349,13 @@ public static ClientProtocol createProxyWithAlignmentContext( boolean withRetries, AtomicBoolean fallbackToSimpleAuth, AlignmentContext alignmentContext) throws IOException { + if (conf.getBoolean(HdfsClientConfigKeys.DFS_RBF_OBSERVER_READ_ENABLE, Review Comment: What was the original behaviour where someone passed in `null` to this function? > Add configuration flag to enable observer reads on routers without using > ObserverReadProxyProvider > -------------------------------------------------------------------------------------------------- > > Key: HDFS-16845 > URL: https://issues.apache.org/jira/browse/HDFS-16845 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Simbarashe Dzinamarira > Assignee: Simbarashe Dzinamarira > Priority: Critical > Labels: pull-request-available > > In order for clients to have routers forward their reads to observers, the > clients must use a proxy with an alignment context. This is currently > achieved by using the ObserverReadProxyProvider. > Using ObserverReadProxyProvider allows backward compatible for client > configurations. > However, the ObserverReadProxyProvider forces an msync on initialization > which is not required with routers. > Performing msync calls is more expensive with routers because the router fans > out the cal to all namespaces, so we'd like to avoid this. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org