[ https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659059#comment-14659059 ]
Andrew Wang commented on HDFS-8833: ----------------------------------- Why does cell size blow up the configuration space? AFAIK values used by other systems are 64KB and 1MB. So it potentially doubles whatever we had before, but I think that still fits within 64 easily, e.g. * Schema: RS * Parameters: (3,2) (6,3) (10,4) * Cell size: 64KB 1MB This is just 6. Even if we add LRC and Hitchhiker later on and that comes to 18. Seems future-proof enough even with just a few bits. > Erasure coding: store EC schema and cell size in INodeFile and eliminate > notion of EC zones > ------------------------------------------------------------------------------------------- > > Key: HDFS-8833 > URL: https://issues.apache.org/jira/browse/HDFS-8833 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode > Affects Versions: HDFS-7285 > Reporter: Zhe Zhang > Assignee: Zhe Zhang > > We have [discussed | > https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754] > storing EC schema with files instead of EC zones and recently revisited the > discussion under HDFS-8059. > As a recap, the _zone_ concept has severe limitations including renaming and > nested configuration. Those limitations are valid in encryption for security > reasons and it doesn't make sense to carry them over in EC. > This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For > simplicity, we should first implement it as an xattr and consider memory > optimizations (such as moving it to file header) as a follow-on. We should > also disable changing EC policy on a non-empty file / dir in the first phase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)