[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-6293:
-----------------------------
Resolution: Fixed
Fix Version/s: 2.5.0
3.0.0
Release Note: Set "dfs.namenode.legacy-oiv-image.dir" to an appropriate
directory to make standby name node or secondary name node save its file system
state in the old fsimage format during checkpointing. This image can be used
for offline analysis using the OfflineImageViewer. Use the "hdfs oiv_legacy"
command to process the old fsimage format.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this to trunk and branch-2. Thanks for discussions, code and
reviews!
> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Priority: Blocker
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
> HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch,
> HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch,
> HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch,
> HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
