[
https://issues.apache.org/jira/browse/HDFS-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948814#comment-13948814
]
Konstantin Shvachko commented on HDFS-6087:
-------------------------------------------
> Currently, block recovery is very complicated during pipeline broken
Indeed the complexity starts when something brakes, and I still don't see how
you propose to simplify the process.
But truncate as discussed in HDFS-3107 is performed on a closed file and
therefore is much less related to pipeline recovery since there is no pipeline
after the file is closed. Which for me makes truncate a simpler task than the
rewrite of the pipeline.
Nicholas in [his
comment|https://issues.apache.org/jira/browse/HDFS-3107?focusedCommentId=13235941&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13235941]
mentioned three ideas to implement truncate. I was recently thinking about
another one.
What if we implement truncate similar to lease recovery. That is when a client
asks to truncate a file, NN changes the list of blocks by deleting some of the
tail ones and decrementing the size of the last. Then NN issues a
DataNodeCommand to recover the last block. DNs as the result of the recovery
will truncated their replica files, and then call commitBockSynchronization()
to report the new length to the NN.
Sorry, didn't want to hijack your jira, so if you intend to proceed with more
general design here I'll re-post my idea under HDFS-3107.
> Unify HDFS write/append/truncate
> --------------------------------
>
> Key: HDFS-6087
> URL: https://issues.apache.org/jira/browse/HDFS-6087
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Reporter: Guo Ruijing
> Attachments: HDFS Design Proposal.pdf, HDFS Design Proposal_3_14.pdf
>
>
> In existing implementation, HDFS file can be appended and HDFS block can be
> reopened for append. This design will introduce complexity including lease
> recovery. If we design HDFS block as immutable, it will be very simple for
> append & truncate. The idea is that HDFS block is immutable if the block is
> committed to namenode. If the block is not committed to namenode, it is HDFS
> client’s responsibility to re-added with new block ID.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
