[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918845#comment-13918845 ] Tsuyoshi OZAWA commented on YARN-1778: -- A log of the test failure is available here: https://builds.apache.org/job/PreCommit-YARN-Build/3234//testReport/ > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918899#comment-13918899 ] Tsuyoshi OZAWA commented on YARN-1778: -- The error message reported on HDFS-6048 is exactly same. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13973413#comment-13973413 ] Wangda Tan commented on YARN-1778: -- I just tried, but cannot reproduce it. Is there anyone else can try to reproduce it too? > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13973879#comment-13973879 ] Tsuyoshi OZAWA commented on YARN-1778: -- I tried but currently I cannot reproduce this problem. [~xgong], should we close this issue once and reopen it when we find to reproduce on trunk? > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297629#comment-14297629 ] Jason Lowe commented on YARN-1778: -- This test is still failing: {noformat} 2015-01-28 22:54:42,813 INFO [main] zookeeper.ZKTestCase (ZKTestCase.java:failed(65)) - FAILED testFSRMStateStoreClientRetry java.lang.AssertionError at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.junit.Assert.assertFalse(Assert.java:74) at org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore.testFSRMStateStoreClientRetry(TestFSRMStateStore.java:289) {noformat} https://builds.apache.org/job/PreCommit-YARN-Build/6442//testReport/ is one example with more details, and I've seen this fail recently on other precommit builds. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303027#comment-14303027 ] zhihai xu commented on YARN-1778: - This test failed randomly. When it failed, the exception is "java.io.IOException: NameNode still not started" I find out the reason for this random error is The NameNode constructor [set started flag|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L780] at the end. {code} this.started.set(true); {code} And it starts [NameNodeRpcServer|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L765] by calling function initialize before started flag is set. initialize=>startCommonServices=>[rpcServer.start|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L639] {code} rpcServer.start(); {code} If the client (which try to call mkdirs) connects to NameNode server before started flag is set, the java.io.IOException: "[NameNode still not started|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java#L1875]"; will happen, then the test will fail. If the client connects to NameNode server after started flag is set, the test will succeed. Although we can fix this error by reordering the code in NameNode constructor: move rpcServer.start to the end before started flag is set, It looks like it is the normal behavior for the name node. It will give exception when the clients connect before started flag is set. since the gap is very small, it will have little side effect and clients can also retry if exception happens. I submitted a patch which add delay before call storeApplicationStateInternal and also not treat this exception(NameNode still not started) as an error. The proof is given in the following logs: {code} 2015-02-03 00:09:18,991 INFO [Thread-2] namenode.NameNode (NameNode.java:(754)) - create NameNode 2015-02-03 00:09:18,991 INFO [Thread-2] namenode.NameNode (NameNode.java:setClientNamenodeAddress(352)) - fs.defaultFS is hdfs://localhost:57792 2015-02-03 00:09:18,991 INFO [Thread-2] namenode.NameNode (NameNode.java:setClientNamenodeAddress(372)) - Clients are to use localhost:57792 to access this namenode/service. 2015-02-03 00:09:18,992 INFO [Thread-2] namenode.NameNode (NameNode.java:(766)) - create NameNode initialize 2015-02-03 00:09:18,996 INFO [Thread-2] hdfs.DFSUtil (DFSUtil.java:httpServerTemplateForNNAndJN(1760)) - Starting Web-server for hdfs at: http://localhost:57791 2015-02-03 00:09:18,997 INFO [Thread-2] http.HttpRequestLog (HttpRequestLog.java:getRequestLog(80)) - Http request log for http.requests.namenode is not defined 2015-02-03 00:09:18,997 INFO [Thread-2] http.HttpServer2 (HttpServer2.java:addGlobalFilter(621)) - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2015-02-03 00:09:18,998 INFO [Thread-2] http.HttpServer2 (HttpServer2.java:addFilter(599)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs 2015-02-03 00:09:18,998 INFO [Thread-2] http.HttpServer2 (HttpServer2.java:addFilter(606)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2015-02-03 00:09:18,999 INFO [Thread-2] http.HttpServer2 (NameNodeHttpServer.java:initWebHdfs(86)) - Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter) 2015-02-03 00:09:18,999 INFO [Thread-2] http.HttpServer2 (HttpServer2.java:addJerseyResourcePackage(525)) - addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/* 2015-02-03 00:09:19,000 INFO [Thread-2] http.HttpServer2 (HttpServer2.java:openListeners(808)) - Jetty bound to port 57791 2015-02-03 00:09:19,000 INFO [Thread-2] mortbay.log (Slf4jLog.java:info(67)) - jetty-6.1.26 2015-02-03 00:09:19,019 INFO [Thread-2] mortbay.log (Slf4jLog.java:info(67)) - Started SelectChannelConnector@localhost:57791 2015-02-03 00:09:19,021 INFO [Thread-2] namenode.FSNamesystem (FSNamesystem.java:(721)) - No KeyProvider found. 2015-02-03 00:09:19,021 INFO [Thread-2] namenode.FSNamesystem (FSNamesystem.java:(731)) - fsLock is fair:true 2015-02-03 00:09:19,022 INFO [Thread-2] blockmanagement.DatanodeManager (DatanodeManager.java:(239)) - dfs.block.invalidate.limit=1000 2015-02-03 00:09:19,022 INFO [Thread-2] blockmanagement.DatanodeManager (DatanodeManager.java:(245)) - dfs.namenode.datanode.registration.ip-hostname-check=t
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303095#comment-14303095 ] Hadoop QA commented on YARN-1778: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696129/YARN-1778.000.patch against trunk revision 8cb4731. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6488//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6488//console This message is automatically generated. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303471#comment-14303471 ] Jason Lowe commented on YARN-1778: -- Thanks for the analysis and patch, [~zxu]! I'm wondering if the test is trying to tell us there really is a problem with FSRMStateStore retries, and therefore fixing the test is actually masking a real problem that needs to be fixed in the main code. If I understand the intent of the test correctly, it's trying to verify that FSRMStateStore will not throw an exception while namenodes are down or coming back up. However if we make the test wait until the namenodes are back up before trying to connect then that defeats most of the point of the test. I think the critical question is: should the "Namenode still not started" exception be retried by either the DFSClient layer or by FSRMStateStore? I think it should, otherwise a client of FSRMStateStore is going to see this exception in a similar, real-world scenario where the Namenode was restarted and wonder why the framework didn't auto-retry. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303547#comment-14303547 ] Tsuyoshi OZAWA commented on YARN-1778: -- I'd like to +1 for retrying at FSRMStateStore layer. We also have an option to add a configuration for setting max number of retry. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303632#comment-14303632 ] zhihai xu commented on YARN-1778: - Hi [~jlowe], that is a very good suggestion. I saw some HDFS error from FSRMStateStore at YARN-2820 which cause RM restart, I already think about doing some retry operation at YARN-2820 last night to make FSRMStateStore better to recover from HDFS error. I will do these change together at YARN-2820. I will make the patch for YARN-2820 in next one or two days. I will resolve this issue in YARN-2820. thanks [~ozawa] for review. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303776#comment-14303776 ] Jason Lowe commented on YARN-1778: -- We may also want to check if this should be handled in DFSClient rather than FSRMStateStore. In other words, is there a reason to have FSRMStateStore make a special case for this, or should all HDFS clients get the benefit of the retry logic? > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303866#comment-14303866 ] zhihai xu commented on YARN-1778: - Hi [~jlowe], thanks for your information. I think HDFS clients retry will have a lot of corner case to cover, it may not be easy to cover all these cases . For example In YARN-2820, we hit the issue:HDFS IOException after HDFS client retry at dfsClient.namenode.complete which is the sub-function(low level) retry in FileSystemRMStateStore#updateFile in the following log. {code} 2014-10-29 23:49:12,202 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore: Updating info for attempt: appattempt_1409135750325_109118_01 at: /tmp/hadoop-yarn/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1409135750325_109118/ appattempt_1409135750325_109118_01 2014-10-29 23:49:19,495 INFO org.apache.hadoop.hdfs.DFSClient: Could not complete /tmp/hadoop-yarn/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1409135750325_109118/ appattempt_1409135750325_109118_01.new.tmp retrying... 2014-10-29 23:49:23,757 INFO org.apache.hadoop.hdfs.DFSClient: Could not complete /tmp/hadoop-yarn/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1409135750325_109118/ appattempt_1409135750325_109118_01.new.tmp retrying... 2014-10-29 23:49:31,120 INFO org.apache.hadoop.hdfs.DFSClient: Could not complete /tmp/hadoop-yarn/yarn/system/rmstore/FSRMStateRoot/RMAppRoot/application_1409135750325_109118/ appattempt_1409135750325_109118_01.new.tmp retrying... 2014-10-29 23:49:46,283 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore: Error updating info for attempt: appattempt_1409135750325_109118_01 java.io.IOException: Unable to close file because the last block does not have enough number of replicas. 2014-10-29 23:49:46,284 ERROR org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Error storing/updating appAttempt: appattempt_1409135750325_109118_01 2014-10-29 23:49:46,916 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received a org.apache.hadoop.yarn.server.resourcemanager.RMFatalEvent of type STATE_STORE_OP_FAILED. {code} HDFS client is low level retry. It doesn't know how the upper layer use it. IMO, It make senses to do the retry in the upper layer for the whole functionality retry, which is similar as doing the retry at different network layers: retry at physical layer, link layer and TCP/IP layer. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306840#comment-14306840 ] zhihai xu commented on YARN-1778: - I submitted a patch YARN-2820.000.patch at YARN-2820 for review. I verified YARN-2820.000.patch can recover from "java.io.IOException: NameNode still not started" with retry, So it will solve this issue. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306866#comment-14306866 ] Tsuyoshi OZAWA commented on YARN-1778: -- [~zxu] cc: [~jlowe] Thank you for the investigation. DFSOutputStream#completeFile includes the logic to retry. It's hard-coded for now: {code} if (retries == 0) { throw new IOException("Unable to close file because the last block" + " does not have enough number of replicas."); } retries--; Thread.sleep(localTimeout); localTimeout *= 2; if (Time.now() - localstart > 5000) { DFSClient.LOG.info("Could not complete " + src + " retrying..."); } {code} How about making these timeouts and number of retries configurable and setting via fs.state-store.num-retries and fs.state-store.retry-interval-ms? It's simpler way to deal with this problem. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308663#comment-14308663 ] zhihai xu commented on YARN-1778: - [~ozawa], Not sure what do you mean. The retries is not hard-coded based on the following code at [DFSOutputStream#completeFile|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L1540] {code} int retries = dfsClient.getConf().nBlockWriteLocateFollowingRetry; {code} nBlockWriteLocateFollowingRetry is decided by configuration "dfs.client.block.write.locateFollowingBlock.retries". The problem for me is the retry in DFSOutputStream#completeFile doesn't work. Based on the log, It retry 5 times in more than 30 seconds and it still doesn't work, then the exception "Unable to close file because the last block does not have enough number of replicas" generated from [FileSystemRMStateStore#writeFile|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java#L583] caused RM restart(). My patch will work better with retry at both high layer(new code) and low layer(old code) because it retry in FileSystemRMStateStore#writeFile, if any exception happen, it will [overwrite the file|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java#L581] and redo everything. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319725#comment-14319725 ] Tsuyoshi OZAWA commented on YARN-1778: -- [~zxu], sorry for confusing you. I meant that we should make the period of retry configurable - it's hard-coded as 5000msec for now. {code} The problem for me is the retry in DFSOutputStream#completeFile doesn't work. {code} How about making the count of retry bigger in FileSystemRMStateStore in startInternal? I think it works well. {code} My patch will work better with retry at both high layer(new code) and low layer(old code) because it retry in FileSystemRMStateStore#writeFile, if any exception happen, it will overwrite the file and redo everything. {code} What kind of failure are you thinking about? I think retrying completeFile here is more straightforward and simple solution. > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323050#comment-14323050 ] zhihai xu commented on YARN-1778: - [~ozawa], That is a good idea. Although we can increase "dfs.client.block.write.locateFollowingBlock.retries" in configuration file and the FileSystemRMStateStore will take the change in startInternal from the configuration file in the following code, it will affect all the other modules to take this change. That may not be feasible. {code} Configuration conf = new Configuration(getConfig()); fs = fsWorkingPath.getFileSystem(conf); {code} To increase the flexibility, we can create a new configuration to customize "dfs.client.block.write.locateFollowingBlock.retries" for FileSystemRMStateStore, which is similar as FS_RM_STATE_STORE_RETRY_POLICY_SPEC to customize "dfs.client.retry.policy.spec" for FileSystemRMStateStore at the following code from startInternal: {code} String retryPolicy = conf.get(YarnConfiguration.FS_RM_STATE_STORE_RETRY_POLICY_SPEC, YarnConfiguration.DEFAULT_FS_RM_STATE_STORE_RETRY_POLICY_SPEC); conf.set("dfs.client.retry.policy.spec", retryPolicy); {code} I will implement a new patch based on this. thanks for the suggestion. zhihai > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[ https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323719#comment-14323719 ] zhihai xu commented on YARN-1778: - Hi [~ozawa], I uploaded a new patch at YARN-2820. Could you review it? thanks > TestFSRMStateStore fails on trunk > - > > Key: YARN-1778 > URL: https://issues.apache.org/jira/browse/YARN-1778 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong >Assignee: zhihai xu > Attachments: YARN-1778.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)