[
https://issues.apache.org/jira/browse/HDFS-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13867122#comment-13867122
]
Todd Lipcon commented on HDFS-5722:
-----------------------------------
Yea, I don't really see the point here. It seems the motivation is a possible
optimization when the NN needs to skip a large section of the image which it
doesn't understand. That's only going to happen on a downgrade scenario, which
is rare and not on a hot path. Plus, do we have examples of _large_ new
sections we plan on adding to the image? Sure, we've added things in the past
like a list of snapshots, but they're typically pretty short. The example of
skipping the entire inodes section seems pretty contrived to me.
HDFS-1435 did show that adding compression slowed down the loading. But that's
because the decompression is on the same thread and the loading is a
single-threaded process. It would really be pretty easy to move the
decompression work onto another core, at which point reading less data is
definitely going to be faster.
Another important factor is the network bandwidth used when one of the image
dirs is on NFS. Many deployments use this for backup. Or, even if the NN isn't
directly writing to NFS, some cron job is backing up the image on a regular
basis using normal OS tools like rsync/scp over the network.
> Implement compression in the HTTP server of SNN / SBN instead of FSImage
> ------------------------------------------------------------------------
>
> Key: HDFS-5722
> URL: https://issues.apache.org/jira/browse/HDFS-5722
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Haohui Mai
>
> The current FSImage format support compression, there is a field in the
> header which specifies the compression codec used to compress the data in the
> image. The main motivation was to reduce the number of bytes to be
> transferred between SNN / SBN / NN.
> The main disadvantage, however, is that it requires the client to access the
> FSImage in strictly sequential order. This might not fit well with the new
> design of FSImage. For example, serializing the data in protobuf allows the
> client to quickly skip data that it does not understand. The compression
> built-in the format, however, complicates the calculation of offsets and
> lengths. Recovering from a corrupted, compressed FSImage is also non-trivial
> as off-the-shelf tools like bzip2recover is inapplicable.
> This jira proposes to move the compression from the format of the FSImage to
> the transport layer, namely, the HTTP server of SNN / SBN. This design
> simplifies the format of FSImage, opens up the opportunity to quickly
> navigate through the FSImage, and eases the process of recovery. It also
> retains the benefits of reducing the number of bytes to be transferred across
> the wire since there are compression on the transport layer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
