Is there any comparison between HBASE-11339 and HDFS-7240? Is their 'Object' a super set of our 'Medium Object'?
2015-05-21 10:38 GMT+08:00 Ted Yu <yuzhih...@gmail.com>: > This is a useful feature, Jon. > > I went over the mega-patch and left some comments on review board. > > I noticed that hbck was not included in the patch. Neither did I find a > sub-task of HBASE-11339 that covers hbck. > > Do you or Jingcheng plan to add MOB-aware capability for hbck ? > > Cheers > > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <j...@cloudera.com> wrote: > > > Hi folks, > > > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O > > and compaction path that allows individual moderately sized values > > (10k-10MB) to be stored so that write amplification is reduced when > > compared to the normal I/O path. At a high level, it provides alternate > > flush and compaction mechanisms that segregates large cells into a > separate > > area where they are not subject to potentially frequent compaction and > > splits that can be encountered in the normal I/O path. A more detailed > > design doc can be found on the hbase-11339 jira. > > > > Jingcheng Du has been working on the mob feature for a while and Anoop, > Ram > > and I have been shepherding him through the design revisions and > > implementation of the feature in the hbase-11339 branch.[2] > > > > The branch we are proposing to merge into master is compatible with > HBase's > > core functionality including snapshots, replication, shell support, > behaves > > well with table alters, bulk loads and does not require external MR > > processes. It has been documented, and subject to many integration test > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection. > > Performance testing of the feature shows what can be a 2x-3x throughput > > improvement for workloads that contain mobs. These results can be seen on > > the hbase 2.0 panel discussion slides from hbasecon (once published). > > > > Recently there have been some hfile encryption related shortcomings that > we > > could address in branch or in master. > > > > Earlier iterations of the feature has been tested in production by users > > that Jingcheng has been responsible for. A version has also been > deployed > > at users I have been responsible for. Some of the folks from Huawei > > (ashutosh) have also been submitting the recent encryption bug reports > > against the hbase-11339 branch so there is some evidence of usage by > them. > > > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the > > feature and feel it is a good time to call a merge vote. Ive posted a > > megapatch version for folks who want to peruse the code. [3] > > > > What do you all think? > > > > Thanks, > > Jingcheng, Jon, Ram, and Anoop. > > > > [1] https://issues.apache.org/jira/browse/HBASE-11339 > > [2] https://github.com/apache/hbase/tree/hbase-11339 > > [3] https://reviews.apache.org/r/34475/ > > -- > > // Jonathan Hsieh (shay) > > // HBase Tech Lead, Software Engineer, Cloudera > > // j...@cloudera.com // @jmhsieh > > >
