[
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567108#comment-15567108
]
Kai Zheng commented on HDFS-10971:
----------------------------------
HSM storage policy is per file, erasure coding policy could also be file
specific, and sound good to support it.
bq. Instead of checking for a "0" or something, we should also add an API like
isErasureCoded to FileStatus. That's more future-proof than encoding special
meaning into the replication factor.
Agree. Otherwise it's not reliable to tell a file is striped/erasurecoded or
not. An API in {{HDFSFileStatus}} isn't enough because some applications don't
want to rely on HDFS client library. To tell a file is replicated or striped is
needed, because some behavior like hflush/hsync isn't supported yet for striped
files so some special handling has to be tweaked particularly. For example, a
case found by [~Sammi] recently, MAPREDUCE-6780.
{{isErasureCoded }} or {{isStriped}}? I'd prefer the former because if we're
lucky to support block level erasure coding, then the API is still in good
sense.
> Distcp should not copy replication factor if source file is erasure coded
> -------------------------------------------------------------------------
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: distcp
> Affects Versions: 3.0.0-alpha1
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if
> {{-pr}} is specified. However, if the source file is EC, the replication
> factor (which is EC policy) should not be replicated to the destination file.
> When a HdfsFileStatus is converted to FileStatus, the replication factor is
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the
> replication factor of an EC file results in an IOException: "Requested
> replication factor of 0 is less than the required minimum of 1 for
> /tmp/dst/dest2"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
