[ https://issues.apache.org/jira/browse/HDFS-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168347#comment-17168347 ]
Chengwei Wang edited comment on HDFS-15493 at 7/31/20, 8:08 AM: ---------------------------------------------------------------- {quote}Therefore setting it to 500 or 1000ms and logging a message each time around the loop should not give any time penalty, but should give us some information about what is happening. {quote} Yes, you are exactly right! The more waiting time and logging would be useful, I would add these. {quote}How long does the shutdown take with the single 4 thread executor? {quote} I just assmued the waiting time was the time cost from `completed loading all INodeDirectory sub-sections` to loading fsimage finished. {code:java} 20/07/31 10:25:59 INFO namenode.FSImageFormatPBINode: Completed loading all INodeDirectory sub-sections 20/07/31 10:26:22 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in 431 seconds. {code} {quote}Are you testing this on the trunk code + this patch, or a different version plus this patch? {quote} I tested this patch on our dev branch which was based on CDH5.10.0 with many patches, the version should be 2.6.0~2.8.0. {quote}Could you try testing 2 executors with 2 threads each? {quote} I had tested this after tested two single thread executors, the time cost was betweent 420s and 430s. I will submit 3 new patches: # one executor with 4 threads with waiting time logging # two single thread executor with waiting time logging and without lock # two fixed 2 thread executors with lock and waiting time logging Let's we test which one would preform best. was (Author: smarthan): {quote}Therefore setting it to 500 or 1000ms and logging a message each time around the loop should not give any time penalty, but should give us some information about what is happening. {quote} Yes, you are exactly right! The more waiting time and logging would be useful, I would add these. {quote}How long does the shutdown take with the single 4 thread executor? {quote} I just assmued the waiting time was the time cost from `completed loading all INodeDirectory sub-sections` to loading fsimage finished. {code:java} 20/07/31 10:25:59 INFO namenode.FSImageFormatPBINode: Completed loading all INodeDirectory sub-sections 20/07/31 10:26:22 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in 431 seconds. {code} {quote}Are you testing this on the trunk code + this patch, or a different version plus this patch? {quote} I tested this patch on our dev branch which was based on CDH5.10.0 with many patches, the version should be 2.6.0~2.8.0. {quote}Could you try testing 2 executors with 2 threads each? {quote} I had tested this after tested two single thread executors, the time cost was betweent 420s and 430s. I will submit 3 new patches: # one executor with 4 threads with waiting time logging # two single thread executor with waiting time logging and without lock # two fixed 2 thread executors with lock and waiting time logging Let's we test which one would preform best. > Update block map and name cache in parallel while loading fsimage. > ------------------------------------------------------------------ > > Key: HDFS-15493 > URL: https://issues.apache.org/jira/browse/HDFS-15493 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Reporter: Chengwei Wang > Priority: Major > Attachments: HDFS-15493.001.patch, HDFS-15493.002.patch, > fsimage-loading.log > > > While loading INodeDirectorySection of fsimage, it will update name cache and > block map after added inode file to inode directory. It would reduce time > cost of fsimage loading to enable these steps run in parallel. > In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load > fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost > reduc to 410s. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org