Not having access cluster for few days, but I will be looking to logs. However, when looking at your logs, it seems that I mispell my mlob dir in the first post. It was "mobdir". The /apps/hbase/data/mobdir/ is nearly empty, sizing 4 or 10 kb
Would this confirm the mob flushing process wouldn't be activated ? 2018-05-19 18:38 GMT+02:00 Ted Yu <[email protected]>: > If you have a chance to look at region server log, you would see some line > such as the following: > > 2018-05-19 16:31:23,548 INFO [MemStoreFlusher.0] regionserver.HMobStore: > Renaming flushed file from > hdfs://mycluster/apps/hbase/data/mobdir/.tmp/ > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb06fb9e800377 > to > hdfs://mycluster/apps/hbase/data/mobdir/data/default/ > IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/ > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb06fb9e800377 > > Meaning Mob store file is first saved under /apps/hbase/data/mobdir/.tmp/ , > then renamed to under the usual location under region directory for the > table. > > From high level, as long as you can query what you ingested, you can be > assured that Mob data is persisted. > > Cheers > > On Sat, May 19, 2018 at 8:43 AM, Nicolas Paris <[email protected]> > wrote: > > > Hi > > > > Yes the data comes back as expected. > > My table is not called "mlob" however since I found such folder > > I thought it was storing mob objects. > > > > I do have 500 folder hashed as you mentionned. They contains the > > whole dataset (2TO) > > However, how beeing sure the data is actually stored as MOB (and not > > as traditional binary) > > > > Thanks > > > > > > 2018-05-19 15:59 GMT+02:00 Ted Yu <[email protected]>: > > > > > bq. look into hdfs hbase/data/mlob > > > > > > Is 'mlob' name of your table ? > > > > > > bq. nearly empty folder > > > > > > Here is listing under a one region table: > > > > > > drwxr-xr-x - hbase hdfs 0 2018-05-16 23:51 > > > /apps/hbase/data/data/default/atlas_janus/.tabledesc > > > drwxr-xr-x - hbase hdfs 0 2018-05-16 23:51 > > > /apps/hbase/data/data/default/atlas_janus/.tmp > > > drwxr-xr-x - hbase hdfs 0 2018-05-17 00:55 > > > /apps/hbase/data/data/default/atlas_janus/ > 8033ea259cb7272d43bc137ca0ab29 > > 06 > > > > > > Not sure if the above matches your description of being nearly empty. > > > Here data is stored under 8033ea259cb7272d43bc137ca0ab2906 > > > > > > If you query the table, does the data come back as expected ? > > > > > > Thanks > > > > > > On Sat, May 19, 2018 at 12:23 AM, Nicolas Paris <[email protected]> > > > wrote: > > > > > > > Hi > > > > > > > > > > > > I am using hbase 1.1 and hive 1.2 > > > > > > > > I created an hbase table with a mob column with the default > > > > threshold (100K) > > > > I mapped the table into hive with a binary format, and loaded > > > > 20M of pdf of size between 50k and 20mb > > > > > > > > Apparently the mob is not populated because when I look into > > > > hdfs hbase/data/mlob, it is a nearly empty folder. > > > > > > > > Does it mean hive cannot populate hbase mob columns ? > > > > > > > > Thanks > > > > > > > > > >
