[ https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795871#comment-17795871 ]
Wei-Chiu Chuang commented on HBASE-28216: ----------------------------------------- I think simpler is always better. In your example, configuring EC policy at family level requires 200k HDFS updates which is going to cause a big disruption to your service for sure. I also don't see the use case to configure EC policy at family level which is too fine grained. Table level EC policy seems like the way to go. > HDFS erasure coding support for table data dirs > ----------------------------------------------- > > Key: HBASE-28216 > URL: https://issues.apache.org/jira/browse/HBASE-28216 > Project: HBase > Issue Type: New Feature > Reporter: Bryan Beaudreault > Priority: Major > > [Erasure > coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html] > (EC) is a hadoop-3 feature which can drastically reduce storage > requirements, at the expense of locality. At my company we have a few hbase > clusters which are extremely data dense and take mostly write traffic, fewer > reads (cold data). We'd like to reduce the cost of these clusters, and EC is > a great way to do that since it can reduce replication related storage costs > by 50%. > It's possible to enable EC policies on sub directories of HDFS. One can > manually set this with {{{}hdfs ec -setPolicy -path > /hbase/data/default/usertable -policy xxxx{}}}. This can work without any > hbase support. > One problem with that is a lack of visibility by operators into which tables > might have EC enabled. I think this is where HBase can help. Here's my > proposal: > * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY > * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set, > verify that the requested policy is available and enabled via > DistributedFileSystem. > getErasureCodingPolicies(). > * During ModifyTableProcedure, add a new state for > MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY. > ** When adding or changing a policy, use DistributedFileSystem. > setErasureCodingPolicy to sync it for the data and archive dir of that table > (or column in table) > ** When removing the property or setting it to empty, use > DistributedFileSystem. > unsetErasureCodingPolicy to remove it from the data and archive dir. > Since this new API is in hadoop-3 only, we'll need to add a reflection > wrapper class for managing the calls and verifying that the API is available. > We'll similarly do that API check in preflightChecks. -- This message was sent by Atlassian Jira (v8.20.10#820010)