[
https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633567#comment-14633567
]
Kai Zheng commented on HDFS-7337:
---------------------------------
Thanks [~andrew.wang] for the thoughts to move on this a bit. They sound good
to me and I'm fine. Some points for further discussion:
bq. ...also the need to persist this information in the fsimage/editlog, ...
Did you mean schema? If so, it looks like a point we all agree with. Per
discussion in HDFS-7859 and related, we planed to do it in follow-on, along
with support of multiple schemas. For now we only support one system defined
schema, RS(6, 3).
bq. Codec enum (e.g. RS, LRC, etc), ...When we get to the point of
fully-pluggable codecs, we can add a special "wildcard" enum value to support
this
Good to have the enum for built-in codecs for now and the wildcard for
customized additional ones in future.
bq. In client's hdfs-site.xml, we can configure a codec implementation for
every codec. This would look something like...
In existing codes we're using the following format for the similar purpose.
Please confirm if it looks good.
{noformat}
/** Raw coder factory for the RS codec. */
public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_KEY =
"io.erasurecode.codec.rs.rawcoder";
/** Raw coder factory for the XOR codec. */
public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_KEY =
"io.erasurecode.codec.xor.rawcoder";
{noformat}
The related codes reside in {{CodecUtil}} reading above configurations. Would
you check it if necessary.
When we're clear what's needed to be done for the phase, I would have an issue
to get them done separately. Thanks.
> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>
> Key: HDFS-7337
> URL: https://issues.apache.org/jira/browse/HDFS-7337
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: Kai Zheng
> Attachments: HDFS-7337-prototype-v1.patch,
> HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
> PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf,
> PluggableErasureCodec.pdf
>
>
> According to HDFS-7285 and the design, this considers to support multiple
> Erasure Codecs via pluggable approach. It allows to define and configure
> multiple codec schemas with different coding algorithms and parameters. The
> resultant codec schemas can be utilized and specified via command tool for
> different file folders. While design and implement such pluggable framework,
> it’s also to implement a concrete codec by default (Reed Solomon) to prove
> the framework is useful and workable. Separate JIRA could be opened for the
> RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation
> to make concrete vendor libraries transparent to the upper layer. This JIRA
> focuses on high level stuffs that interact with configuration, schema and etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
