[
https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Plamen Jeliazkov updated HDFS-3107:
-----------------------------------
Attachment: HDFS-3107.patch
Attaching an initial patch for discussion.
I looked at the PDFs by [~lei_chang] and I do not see this breaking the API and
ideas proposed there.
Konstantin's comment above with the steps of truncate are how this patch works.
Some additional notes I'd like to make here about the patch:
# I am making use of FsDatasetImpl's truncateBlock() method for performing the
actual truncation. That method does not appear to do copy-on-write; it just
truncates the original block.
# When truncation is completed on the DNs it is reported thru the primary DN's
commitBlockSynchronization call.
# The file length on the INodeFile will change with the truncate call.
I think we will be able to support snapshots if I can keep the old last blocks
as well as any truncated data around. Currently, I can see us keeping the old
blocks around (if truncate is done on block-boundary), but I do not have a way
to keep the actual truncated data around. Probably more work needs to be done
on FsDatasetImpl and FSNamesystem side. As [~szetszwo] suggested, a full design
will be required.
Let me know what you guys think.
I appreciate all feedback and will try to incorporate it carefully.
> HDFS truncate
> -------------
>
> Key: HDFS-3107
> URL: https://issues.apache.org/jira/browse/HDFS-3107
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Reporter: Lei Chang
> Assignee: Plamen Jeliazkov
> Attachments: HDFS-3107.patch, HDFS_truncate_semantics_Mar15.pdf,
> HDFS_truncate_semantics_Mar21.pdf
>
> Original Estimate: 1,344h
> Remaining Estimate: 1,344h
>
> Systems with transaction support often need to undo changes made to the
> underlying storage when a transaction is aborted. Currently HDFS does not
> support truncate (a standard Posix operation) which is a reverse operation of
> append, which makes upper layer applications use ugly workarounds (such as
> keeping track of the discarded byte range per file in a separate metadata
> store, and periodically running a vacuum process to rewrite compacted files)
> to overcome this limitation of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
