[
https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020574#comment-16020574
]
Andrew Wang commented on HDFS-7337:
-----------------------------------
Hi folks, thanks for the discussion,
bq. Is there a way to choose a system-wide default codec? So that after the
cluster being initialized, users and admins can just specify a zone / directory
to be "erasure coded", instead of choosing from several different codes, and
each one has its own trade-offs, which require user / admin to understand?
We had a system default policy originally, but then moved away from it. I'm
open to bringing it back if we believe that there's typically only one policy
in a cluster. I think this is likely true.
bq. My concern is that, even if the admin is able to add policy via the API
dynamically, it still requires the admin to reboot NN, or ssh into NN / change
conf files and reload NN confs, to enable the policy? It makes the workflow
complicated.
Yea, this is true. I can envision how this would work with just CLI commands:
add/remove/enable/disable. I don't know how we'd do this with just config,
since we want the safety of persisting things in the fsimage.
So, shall we do it all via API?
> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>
> Key: HDFS-7337
> URL: https://issues.apache.org/jira/browse/HDFS-7337
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
> Reporter: Zhe Zhang
> Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-7337-prototype-v1.patch,
> HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
> PluggableErasureCodec.pdf, PluggableErasureCodec-v2.pdf,
> PluggableErasureCodec-v3.pdf, PluggableErasureCodec v4.pdf
>
>
> According to HDFS-7285 and the design, this considers to support multiple
> Erasure Codecs via pluggable approach. It allows to define and configure
> multiple codec schemas with different coding algorithms and parameters. The
> resultant codec schemas can be utilized and specified via command tool for
> different file folders. While design and implement such pluggable framework,
> it’s also to implement a concrete codec by default (Reed Solomon) to prove
> the framework is useful and workable. Separate JIRA could be opened for the
> RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation
> to make concrete vendor libraries transparent to the upper layer. This JIRA
> focuses on high level stuffs that interact with configuration, schema and etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
