[
https://issues.apache.org/jira/browse/HDFS-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920929#action_12920929
]
Lu Yilei commented on HDFS-1435:
--------------------------------
to Kuang, If the fsimage is very big. The network is full in a short time when
SeconaryNamenode do checkpoint, leading to Jobtracker access Namenode to get
relevant file data to fail in job initialization phase. So we limit
transmission speed and compress transmission to resolve the problem.
We have complete the development and testing. But we not add the code that
check NameNode fsimage and SecondaryNameNode when download CheckpointFiles.
Because it may lead to other risks.
Please see the patch that limit transmission speed and compress transmission.
Next I will contribute other patch that check NameNode fsimage and
SecondaryNameNode when download CheckpointFiles. Thanks.
> Provide an option to store fsimage compressed
> ---------------------------------------------
>
> Key: HDFS-1435
> URL: https://issues.apache.org/jira/browse/HDFS-1435
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: trunkImageCompress.patch, trunkImageCompress1.patch
>
>
> Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network
> bandwidth when secondary NN uploads a new fsimage to primary NN.
> If we could store fsimage compressed, the problem could be greatly alleviated.
> I plan to provide a new configuration hdfs.image.compressed with a default
> value of false. If it is set to be true, fsimage is stored as compressed.
> The fsimage will have a new layout with a new field "compressed" in its
> header, indicating if the namespace is stored compressed or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
