2018-05-19 20:08 GMT+02:00 Ted Yu <[email protected]>: > Mob store file is renamed from /apps/hbase/data/mobdir to the final > location under region directory for the table. > > This explains why you don't see much data under mobdir since data ingestion > has finished. >
Well, I monitored the mobdir folder during ingestion. Nothing happened in it. Data were directly going under the table in the region. BTW, thats why I was thinking the pdf were treated as regular binary files. Certainly reading the regionserver logs will help. Finallly, if hive is able to load mob columns, that's a good news for me. > > Cheers > > On Sat, May 19, 2018 at 9:58 AM, Nicolas Paris <[email protected]> > wrote: > > > Not having access cluster for few days, but I will be looking > > to logs. > > > > However, when looking at your logs, it seems that I mispell > > my mlob dir in the first post. It was "mobdir". > > The /apps/hbase/data/mobdir/ is nearly empty, sizing 4 or 10 kb > > > > Would this confirm the mob flushing process wouldn't be activated ? > > > > > > > > 2018-05-19 18:38 GMT+02:00 Ted Yu <[email protected]>: > > > > > If you have a chance to look at region server log, you would see some > > line > > > such as the following: > > > > > > 2018-05-19 16:31:23,548 INFO [MemStoreFlusher.0] > regionserver.HMobStore: > > > Renaming flushed file from > > > hdfs://mycluster/apps/hbase/data/mobdir/.tmp/ > > > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb > 06fb9e800377 > > > to > > > hdfs://mycluster/apps/hbase/data/mobdir/data/default/ > > > IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/ > > > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb > 06fb9e800377 > > > > > > Meaning Mob store file is first saved under > > /apps/hbase/data/mobdir/.tmp/ , > > > then renamed to under the usual location under region directory for the > > > table. > > > > > > From high level, as long as you can query what you ingested, you can be > > > assured that Mob data is persisted. > > > > > > Cheers > > > > > > On Sat, May 19, 2018 at 8:43 AM, Nicolas Paris <[email protected]> > > > wrote: > > > > > > > Hi > > > > > > > > Yes the data comes back as expected. > > > > My table is not called "mlob" however since I found such folder > > > > I thought it was storing mob objects. > > > > > > > > I do have 500 folder hashed as you mentionned. They contains the > > > > whole dataset (2TO) > > > > However, how beeing sure the data is actually stored as MOB (and not > > > > as traditional binary) > > > > > > > > Thanks > > > > > > > > > > > > 2018-05-19 15:59 GMT+02:00 Ted Yu <[email protected]>: > > > > > > > > > bq. look into hdfs hbase/data/mlob > > > > > > > > > > Is 'mlob' name of your table ? > > > > > > > > > > bq. nearly empty folder > > > > > > > > > > Here is listing under a one region table: > > > > > > > > > > drwxr-xr-x - hbase hdfs 0 2018-05-16 23:51 > > > > > /apps/hbase/data/data/default/atlas_janus/.tabledesc > > > > > drwxr-xr-x - hbase hdfs 0 2018-05-16 23:51 > > > > > /apps/hbase/data/data/default/atlas_janus/.tmp > > > > > drwxr-xr-x - hbase hdfs 0 2018-05-17 00:55 > > > > > /apps/hbase/data/data/default/atlas_janus/ > > > 8033ea259cb7272d43bc137ca0ab29 > > > > 06 > > > > > > > > > > Not sure if the above matches your description of being nearly > empty. > > > > > Here data is stored under 8033ea259cb7272d43bc137ca0ab2906 > > > > > > > > > > If you query the table, does the data come back as expected ? > > > > > > > > > > Thanks > > > > > > > > > > On Sat, May 19, 2018 at 12:23 AM, Nicolas Paris < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > Hi > > > > > > > > > > > > > > > > > > I am using hbase 1.1 and hive 1.2 > > > > > > > > > > > > I created an hbase table with a mob column with the default > > > > > > threshold (100K) > > > > > > I mapped the table into hive with a binary format, and loaded > > > > > > 20M of pdf of size between 50k and 20mb > > > > > > > > > > > > Apparently the mob is not populated because when I look into > > > > > > hdfs hbase/data/mlob, it is a nearly empty folder. > > > > > > > > > > > > Does it mean hive cannot populate hbase mob columns ? > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > >
