[
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15566111#comment-15566111
]
Zhe Zhang commented on HDFS-10971:
----------------------------------
Thanks for the discussions Andrew / Wei-Chiu.
For the initial 3.0 release I prefer a simpler strategy of only supporting
{{-pr}} when both the source and destination directories are replicated.
{quote}
-setPolicy \[-p <policyName>\] <path>
Sets an ErasureCoding policy on a directory at the specified path.
path: An directory in HDFS. This is a mandatory parameter. Setting a policy
only affects newly created files, and does not affect existing files.
{quote}
The current EC policy protocol is that if a directory is configured with a
policy, it will be applied to all newly created direct children files under it.
Extending this protocol will complicate how users/admins understand EC
policies, and probably needs more discussions (maybe after the EC feature has
been used in production for a while).
Thoughts?
> Distcp should not copy replication factor if source file is erasure coded
> -------------------------------------------------------------------------
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: distcp
> Affects Versions: 3.0.0-alpha1
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if
> {{-pr}} is specified. However, if the source file is EC, the replication
> factor (which is EC policy) should not be replicated to the destination file.
> When a HdfsFileStatus is converted to FileStatus, the replication factor is
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the
> replication factor of an EC file results in an IOException: "Requested
> replication factor of 0 is less than the required minimum of 1 for
> /tmp/dst/dest2"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
