[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988435#comment-13988435
]
Suresh Srinivas commented on HDFS-6293:
---------------------------------------
bq. PB would be preferable to JSON. I'd be interested to hear your reasoning
why JSON is significantly easier; I figured since we already have PB in the
build and experience using it, it wouldn't be that much work.
PB implementation for large number of objects as an array has read side issues
and requires designing the protobuf more carefully and investment of time.
[~wheat9] understands this better and can answer (or you can see the structure
of current fsimage proto, where this is considered). If you want to pursue that
direction you are welcome. You can add that by adding a new configuration for
configuring output format. Doing it in JSON gives us a quick solution and is
sufficient for use cases we are looking for.
bq. Can we provide some kind of REST API for fetching this extra listing file?
This is preferable to manually finding the file and doing scp.
Good idea. Lets do it in another jira. Please create a related jira.
bq. What kinds of atomicity guarantees are there between the fsimage and this
listing? We'd like to be able to take the listing and replay the edit log on
top. Including the txid in the listing is also important for this work.
Not sure what your question means. This report has the same state as fsimage,
given it is done right after the checkpoint. The printed report would include
transaction id information.
bq. Will this also be done by other saveNamespaces besides checkpointing (i.e.
"-saveNamespace" as well as at startup)?
Not sure if that is necessary. If it is, we can certainly add that in another
jira. One thing that I should have mentioned is, currently this file exists
only in standby and will not be shipped to active.
bq. I'd also appreciate if you posted any further call-ins to this JIRA
Certainly, where possible. But you have all the information in the jira and
have an opportunity to discuss it, right?
> Issues with OIV processing PB-based fsimages
> --------------------------------------------
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Kihwal Lee
> Priority: Blocker
> Attachments: Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
