[
https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Yao updated HADOOP-13200:
-----------------------------
Release Note:
CodecRegistry uses ServiceLoader to dynamically load all implementations of
RawErasureCoderFactory. In Hadoop 3.0, there are several built-in
implementations, and user can also provide self-defined implementations with
the corresponding resource files.
For each codec, user can configure the order of the implementations with the
configuration keys:
`io.erasurecode.codec.rs.rawcoders` for the default RS codec,
`io.erasurecode.codec.rs-legacy.rawcoders` for the legacy RS codec,
`io.erasurecode.codec.xor.rawcoders` for the XOR codec.
User can also configure self-defined codec with the configuration key like:
`io.erasurecode.codec.self-defined.rawcoders`.
For each codec, Hadoop will use the implementation according to the order
configured. If the former implementation fails, it will fall back to call the
latter one. The order is defined by a list of coder names separated by commas.
The names for the built-in implementations are:
`rs_native` and `rs_java` for the default RS codec, of which the former is a
native implementation which leverages Intel ISA-L library, which is the default
implementation and the latter is the implementation in pure Java,
`rs-legacy_java` for the legacy RS codec, which is the default implementation
in pure Java,
`xor_native` and `xor_java` for the XOR codec, of which the former is the Intel
ISA-L implementation which is the default one and the latter in pure Java.
was:
CodecRegistry uses ServiceLoader to dynamically load all implementations of
RawErasureCoderFactory. In Hadoop 3.0, there are several built-in
implementations, and user can also provide self-defined implementations with
the corresponding resource files.
For each codec, user can configure the order of the implementations with the
configuration keys:
`io.erasurecode.codec.rs.rawcoders` for the default RS codec,
`io.erasurecode.codec.rs-legacy.rawcoders` for the legacy RS codec,
`io.erasurecode.codec.rs.rawcoders` for the XOR codec.
User can also configure self-defined codec with the configuration key like:
`io.erasurecode.codec.self-defined.rawcoders`.
For each codec, Hadoop will use the implementation according to the order
configured. If the former implementation fails, it will fall back to call the
latter one. The order is defined by a list of coder names separated by commas.
The names for the built-in implementations are:
`rs_native` and `rs_java` for the default RS codec, of which the former is a
native implementation which leverages Intel ISA-L library, which is the default
implementation and the latter is the implementation in pure Java,
`rs-legacy_java` for the legacy RS codec, which is the default implementation
in pure Java,
`xor_native` and `xor_java` for the XOR codec, of which the former is the Intel
ISA-L implementation which is the default one and the latter in pure Java.
> Implement customizable and configurable erasure coders
> ------------------------------------------------------
>
> Key: HADOOP-13200
> URL: https://issues.apache.org/jira/browse/HADOOP-13200
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Kai Zheng
> Assignee: Tim Yao
> Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Fix For: 3.0.0-alpha3
>
> Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch,
> HADOOP-13200.04.patch, HADOOP-13200.05.patch, HADOOP-13200.06.patch,
> HADOOP-13200.07.patch, HADOOP-13200.08.patch, HADOOP-13200.09.patch,
> HADOOP-13200.10.patch, HADOOP-13200.11.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may
> be some better approach allowing to customize and configure erasure coders
> than the current having raw coder factory, as [~cmccabe] suggested. Will copy
> the relevant comments here to continue the discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
