[
https://issues.apache.org/jira/browse/HDFS-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hairong Kuang updated HDFS-1435:
--------------------------------
Attachment: trunkImageCompress.patch
This patch changed the fsimage's format to support compression. The third field
in the header (isCompressed) indicates if the image is stored as compressed. If
yes, the fourth field stores the compression codec.
The HDFS admin could configure if s/he wants to store fsimage compressed and
which codec is used to compress its fsimage. The codec to be used for storing
or reading an fsimage has to be one of the codecs specified in
io.compression.codecs or be either GzipCodec or DefaultCodec if
io.compression.codecs is not configured.
> Provide an option to store fsimage compressed
> ---------------------------------------------
>
> Key: HDFS-1435
> URL: https://issues.apache.org/jira/browse/HDFS-1435
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: trunkImageCompress.patch
>
>
> Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network
> bandwidth when secondary NN uploads a new fsimage to primary NN.
> If we could store fsimage compressed, the problem could be greatly alleviated.
> I plan to provide a new configuration hdfs.image.compressed with a default
> value of false. If it is set to be true, fsimage is stored as compressed.
> The fsimage will have a new layout with a new field "compressed" in its
> header, indicating if the namespace is stored compressed or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
