[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984353#comment-13984353
]
Kihwal Lee commented on HDFS-6293:
----------------------------------
bq. One solution we can do is - Add an option to print directory tree
information (along the lines ls -r) that works against fsimage.
It is still there in 2.4. It was removed by HDFS-6164 in 2.5. Even if we add
it back, the excessive memory requirement makes it useless.
bq. We could also consider either building a tool that works efficiently in
memory or reorganize the fsimage to make that possible.
I will be great if someone can come up with a standalone tool that allows
dumping directory structure and content with, say, 1-2GB heap AND completes in
comparable execution time. Otherwise, we will have to rearrange the internal
layout of fsimage similar to the previous format.
bq. Kihwal Lee, can you please provide the use cases you are using OIV for?
There is existing apps that use a custom Visitor similar to lsr. It outputs
directory entries with full path and list of blocks for files.
> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Kihwal Lee
> Priority: Blocker
> Attachments: Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
