[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996442#comment-13996442 ]
Kihwal Lee commented on HDFS-6293: ---------------------------------- Here is recap. * oiv_legacy for debugging and inspecting old fsimages. As long as NN is capable of reading and upgrading from it, we should keep it. * NN's ability to save in old fsimage format: This should go away once we have a more sustainable solution. Postprocessing ([~wheat9] 's proposal) will cover many use cases. We may need something else. This is to be determined. * Keep adding features to the old fsimage format: Please don't! * Changing the layout of the new PB-based fsimage back to the old way: Heck no! We will deprecate NN's ability to save in the old fsimage format, once we agree on what the long term solution is. We should create a separate JIRA and continue discussion. As [~sureshms] suggested, I will prepare the patch for trunk as well with [~ajisakaa]'s comment incorporated. > Issues with OIV processing PB-based fsimages > -------------------------------------------- > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 2.4.0 > Reporter: Kihwal Lee > Assignee: Haohui Mai > Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)