[
https://issues.apache.org/jira/browse/HDFS-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168347#comment-17168347
]
Chengwei Wang edited comment on HDFS-15493 at 7/31/20, 8:08 AM:
----------------------------------------------------------------
{quote}Therefore setting it to 500 or 1000ms and logging a message each time
around the loop should not give any time penalty, but should give us some
information about what is happening.
{quote}
Yes, you are exactly right! The more waiting time and logging would be useful,
I would add these.
{quote}How long does the shutdown take with the single 4 thread executor?
{quote}
I just assmued the waiting time was the time cost from `completed loading all
INodeDirectory sub-sections` to loading fsimage finished.
{code:java}
20/07/31 10:25:59 INFO namenode.FSImageFormatPBINode: Completed loading all
INodeDirectory sub-sections
20/07/31 10:26:22 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in 431
seconds.
{code}
{quote}Are you testing this on the trunk code + this patch, or a different
version plus this patch?
{quote}
I tested this patch on our dev branch which was based on CDH5.10.0 with many
patches, the version should be 2.6.0~2.8.0.
{quote}Could you try testing 2 executors with 2 threads each?
{quote}
I had tested this after tested two single thread executors, the time cost was
betweent 420s and 430s.
I will submit 3 new patches:
# one executor with 4 threads with waiting time logging
# two single thread executor with waiting time logging and without lock
# two fixed 2 thread executors with lock and waiting time logging
Let's we test which one would preform best.
was (Author: smarthan):
{quote}Therefore setting it to 500 or 1000ms and logging a message each time
around the loop should not give any time penalty, but should give us some
information about what is happening.
{quote}
Yes, you are exactly right! The more waiting time and logging would be useful,
I would add these.
{quote}How long does the shutdown take with the single 4 thread executor?
{quote}
I just assmued the waiting time was the time cost from `completed loading all
INodeDirectory sub-sections` to loading fsimage finished.
{code:java}
20/07/31 10:25:59 INFO namenode.FSImageFormatPBINode: Completed loading all
INodeDirectory sub-sections
20/07/31 10:26:22 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in 431
seconds.
{code}
{quote}Are you testing this on the trunk code + this patch, or a different
version plus this patch?
{quote}
I tested this patch on our dev branch which was based on CDH5.10.0 with many
patches, the version should be 2.6.0~2.8.0.
{quote}Could you try testing 2 executors with 2 threads each?
{quote}
I had tested this after tested two single thread executors, the time cost was
betweent 420s and 430s.
I will submit 3 new patches:
# one executor with 4 threads with waiting time logging
# two single thread executor with waiting time logging and without lock
# two fixed 2 thread executors with lock and waiting time logging
Let's we test which one would preform best.
> Update block map and name cache in parallel while loading fsimage.
> ------------------------------------------------------------------
>
> Key: HDFS-15493
> URL: https://issues.apache.org/jira/browse/HDFS-15493
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Chengwei Wang
> Priority: Major
> Attachments: HDFS-15493.001.patch, HDFS-15493.002.patch,
> fsimage-loading.log
>
>
> While loading INodeDirectorySection of fsimage, it will update name cache and
> block map after added inode file to inode directory. It would reduce time
> cost of fsimage loading to enable these steps run in parallel.
> In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load
> fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost
> reduc to 410s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
