[
https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937412#comment-15937412
]
Andrew Wang commented on HDFS-7337:
-----------------------------------
Hi Kai,
bq. Ah yes, it's a good point to refactor out the rs-legacy codec, not listing
the related policies as built-in but would rather add it back by pluggable. I
thought this is what you meant, right? Totally make sense if so.
Yep, exactly :)
bq. by writing an XML file, admin users can define their own codecs, schemas
and policies.
I gave the v3 doc a quick look, it sounds like the XML file is basically an
input for a "refresh" command, and is unnecessary after it's loaded since the
information is persisted to the NN metadata.
It might be simpler for admins if we still do this over an RPC interface.
Rather than specifying all the ECSchema info as arguments, the CLI tool can
take the XML file as input. The CLI tool can also perform basic validation, and
prompt the user when doing possibly destructive operations like removing a
schema.
I like this a bit better since the admin doesn't need to be SSH'd into the
NameNode, know where to put the XML file, or know which NN is active. It might
also simplify error reporting for malformed requests, since it'll be returned
on the CLI rather than in a log file.
> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>
> Key: HDFS-7337
> URL: https://issues.apache.org/jira/browse/HDFS-7337
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
> Reporter: Zhe Zhang
> Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-7337-prototype-v1.patch,
> HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
> PluggableErasureCodec.pdf, PluggableErasureCodec-v2.pdf,
> PluggableErasureCodec-v3.pdf
>
>
> According to HDFS-7285 and the design, this considers to support multiple
> Erasure Codecs via pluggable approach. It allows to define and configure
> multiple codec schemas with different coding algorithms and parameters. The
> resultant codec schemas can be utilized and specified via command tool for
> different file folders. While design and implement such pluggable framework,
> it’s also to implement a concrete codec by default (Reed Solomon) to prove
> the framework is useful and workable. Separate JIRA could be opened for the
> RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation
> to make concrete vendor libraries transparent to the upper layer. This JIRA
> focuses on high level stuffs that interact with configuration, schema and etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
