maobaolong created HDFS-13269: --------------------------------- Summary: After too many open file exception occurred, the standby NN never do checkpoint Key: HDFS-13269 URL: https://issues.apache.org/jira/browse/HDFS-13269 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 3.2.0 Reporter: maobaolong
do saveNameSpace in dfsadmin. The output as following: {code:java} saveNamespace: No image directories available! {code} The Namenode log show: {code:java} [2018-01-13T10:32:19.903+08:00] [INFO] [Standby State Checkpointer] : Triggering checkpoint because there have been 10159265 txns since the last checkpoint, which exceeds the configured threshold 10000000 [2018-01-13T10:32:19.903+08:00] [INFO] [Standby State Checkpointer] : Save namespace ... ... [2018-01-13T10:37:10.539+08:00] [WARN] [1985938863@qtp-61073295-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@HOST_A:50070] : EXCEPTION java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:686) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) [2018-01-13T10:37:15.421+08:00] [ERROR] [FSImageSaver for /data0/nn of type IMAGE_AND_EDITS] : Unable to save image for /data0/nn java.io.FileNotFoundException: /data0/nn/current/fsimage_0000000040247283317.md5.tmp (Too many open files) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.hadoop.hdfs.util.AtomicFileOutputStream.<init>(AtomicFileOutputStream.java:58) at org.apache.hadoop.hdfs.util.MD5FileUtils.saveMD5File(MD5FileUtils.java:157) at org.apache.hadoop.hdfs.util.MD5FileUtils.saveMD5File(MD5FileUtils.java:149) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:990) at org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:1039) at java.lang.Thread.run(Thread.java:745) [2018-01-13T10:37:15.421+08:00] [ERROR] [Standby State Checkpointer] : Error reported on storage directory Storage Directory /data0/nn [2018-01-13T10:37:15.421+08:00] [WARN] [Standby State Checkpointer] : About to remove corresponding storage: /data0/nn [2018-01-13T10:37:15.429+08:00] [ERROR] [Standby State Checkpointer] : Exception in doCheckpoint java.io.IOException: Failed to save in any storage directories while saving namespace. at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1176) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:1107) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(StandbyCheckpointer.java:185) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.access$1400(StandbyCheckpointer.java:62) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.doWork(StandbyCheckpointer.java:353) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.access$700(StandbyCheckpointer.java:260) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread$1.run(StandbyCheckpointer.java:280) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.run(StandbyCheckpointer.java:276) ... [2018-01-13T15:52:33.783+08:00] [INFO] [Standby State Checkpointer] : Save namespace ... [2018-01-13T15:52:33.783+08:00] [ERROR] [Standby State Checkpointer] : Exception in doCheckpoint java.io.IOException: No image directories available! at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1152) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:1107) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(StandbyCheckpointer.java:185) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.access$1400(StandbyCheckpointer.java:62) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.doWork(StandbyCheckpointer.java:353) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.access$700(StandbyCheckpointer.java:260) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread$1.run(StandbyCheckpointer.java:280) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.run(StandbyCheckpointer.java:276) {code} After seeing the NNStorage#reportErrorsOnDirectory {code:java} private void reportErrorsOnDirectory(StorageDirectory sd) { LOG.error("Error reported on storage directory {}", sd); if(LOG.isDebugEnabled()){ String lsd = listStorageDirectories(); LOG.debug("current list of storage dirs:{}", lsd); } LOG.warn("About to remove corresponding storage: {}", sd.getRoot() .getAbsolutePath()); try { sd.unlock(); } catch (Exception e) { LOG.warn("Unable to unlock bad storage directory: {}", sd.getRoot() .getPath(), e); } if (getStorageDirs().remove(sd)) { this.removedStorageDirs.add(sd); } if(LOG.isDebugEnabled()){ String lsd = listStorageDirectories(); LOG.debug("at the end current list of storage dirs:{}", lsd); } } {code} I think when the FileNotFoundException(Too many open file) occurred, we should not remove the storageDirs, because the exception can be cure after several minute. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org