[ https://issues.apache.org/jira/browse/HDFS-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381949#comment-14381949 ]
Hudson commented on HDFS-7928: ------------------------------ FAILURE: Integrated in Hadoop-Hdfs-trunk #2076 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2076/]) HDFS-7928. Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy. Contributed by Rushabh Shah. (kihwal: rev fc1031af749435dc95efea6745b1b2300ce29446) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockListAsLongs.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/UpgradeUtilities.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java > Scanning blocks from disk during rolling upgrade startup takes a lot of time > if disks are busy > ---------------------------------------------------------------------------------------------- > > Key: HDFS-7928 > URL: https://issues.apache.org/jira/browse/HDFS-7928 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Affects Versions: 2.6.0 > Reporter: Rushabh S Shah > Assignee: Rushabh S Shah > Fix For: 2.8.0 > > Attachments: HDFS-7928-v1.patch, HDFS-7928-v2.patch, HDFS-7928.patch > > > We observed this issue in rolling upgrade to 2.6.x on one of our cluster. > One of the disks was very busy and it took long time to scan that disk > compared to other disks. > Seeing the sar (System Activity Reporter) data we saw that the particular > disk was very busy performing IO operations. > Requesting for an improvement during datanode rolling upgrade. > During shutdown, we can persist the whole volume map on the disk and let the > datanode read that file and create the volume map during startup after > rolling upgrade. > This will not require the datanode process to scan all the disk and read the > block. > This will significantly improve the datanode startup time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)