[
https://issues.apache.org/jira/browse/HBASE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16307602#comment-16307602
]
Mike Drob edited comment on HBASE-19369 at 1/2/18 2:09 AM:
-----------------------------------------------------------
I'm not sure of the underlying technical reasons, all I have are the EC docs.
http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Limitations
{quote}
Certain HDFS file write operations, i.e., hflush, hsync and append, are not
supported on erasure coded files due to substantial technical challenges.
* append() on an erasure coded file will throw IOException.
* hflush() and hsync() on DFSStripedOutputStream are no-op. Thus calling
hflush() or hsync() on an erasure coded file can not guarantee data being
persistent.
A client can use StreamCapabilities API to query whether a OutputStream
supports hflush() and hsync(). If the client desires data persistence via
hflush() and hsync(), the current remedy is creating such files as regular 3x
replication files in a non-erasure-coded directory, or using
FSDataOutputStreamBuilder#replicate() API to create 3x replication files in an
erasure-coded directory.
{quote}
was (Author: mdrob):
I'm not sure of the underlying technical reasons, all I have are the EC docs.
http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Limitations
{quote}
Certain HDFS file write operations, i.e., hflush, hsync and append, are not
supported on erasure coded files due to substantial technical challenges.
* append() on an erasure coded file will throw IOException.
* hflush() and hsync() on DFSStripedOutputStream are no-op. Thus calling
hflush() or hsync() on an erasure coded file can not guarantee data being
persistent.
A client can use StreamCapabilities API to query whether a OutputStream
supports hflush() and hsync(). If the client desires data persistence via
hflush() and hsync(), the current remedy is creating such files as regular 3x
replication files in a non-erasure-coded directory, or using
FSDataOutputStreamBuilder#replicate() API to create 3x replication files in an
erasure-coded directory.
{quote}
> HBase Should use Builder Pattern to Create Log Files while using WAL on
> Erasure Coding
> --------------------------------------------------------------------------------------
>
> Key: HBASE-19369
> URL: https://issues.apache.org/jira/browse/HBASE-19369
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Alex Leblang
> Assignee: Alex Leblang
> Attachments: HBASE-19369.master.001.patch,
> HBASE-19369.master.002.patch, HBASE-19369.master.003.patch,
> HBASE-19369.master.004.patch, HBASE-19369.v5.patch, HBASE-19369.v6.patch,
> HBASE-19369.v7.patch, HBASE-19369.v8.patch
>
>
> Right now an HBase instance using the WAL won't function properly in an
> Erasure Coded environment. We should change the following line to use the
> hdfs.DistributedFileSystem builder pattern
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java#L92
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
