On Thu, May 28, 2015 at 7:53 PM, Ted Yu <[email protected]> wrote: > What release of hbase are you using ? > I'm using 0.98.9. I should have mentioned it before anything else.
> > If you trace 5fe111d3302cefb2e96d541a489ff026 in master log, do you find > some clue ? > >From the timestamp of the logs, all these warnings are given out when I do "pio status" which tests if hbase is up by writing to it a table "predictionio_eventdata:events_0" then deleting it. > > Pastebin related log snippet if possible. > http://pastebin.com/WDV01WQ7. There is only one line of "WARN No HRegionInfo" instead of thousands of lines since I issued the "hbase hbck -fixEmptyMetaCells" just now (but it keeps accumulating and that causes problem). Also one thing I notice is that before fixing the meta table (-fixEmptyMetaCells), in hbase shell, counting the non-existing table 'predictionio_eventdata:events_0' takes much longer (a minute or two) than some other non-existing tables. It spits out this line (or more) before prompting that the table does not exist: ERROR: HRegionInfo was null or empty in hbase:meta, row=keyvalues={predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:seqnumDuringOpen/1432926325684/Put/vlen=8/mvcc=0, predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:server/1432926325684/Put/vlen=46/mvcc=0, predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:serverstartcode/1432926325684/Put/vlen=8/mvcc=0} Tom > > Thanks > > On Thu, May 28, 2015 at 6:48 PM, Tom Chan <[email protected]> wrote: > > > Hi, > > > > I kept running into this whenever I remove tables > > > > WARN catalog.MetaReader: No serialized HRegionInfo in > > > > > keyvalues={predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:seqnumDuringOpen/1430338711901/Put/vlen=8/mvcc=0, > > > > > predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:server/1430338711901/Put/vlen=46/mvcc=0, > > > > > predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:serverstartcode/1430338711901/Put/vlen=8/mvcc=0} > > > > create tables > > > > WARN client.HBaseAdmin: No serialized HRegionInfo in > > > > > keyvalues={predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:seqnumDuringOpen/1427845220983/Put/vlen=8/mvcc=0, > > > > > predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:server/1427845220983/Put/vlen=46/mvcc=0, > > > > > predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:serverstartcode/1427845220983/Put/vlen=8/mvcc=0} > > > > > > When doing hbck, I found some 1800 inconsistencies of the form > > > > ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta > > > > > > > keyvalues={predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:seqnumDuringOpen/1431758974883/Put/vlen=8/mvcc=0, > > > > > predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:server/1431758974883/Put/vlen=46/mvcc=0, > > > > > predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:serverstartcode/1431758974883/Put/vlen=8/mvcc=0} > > > > while all the tables are okay. > > > > I ran "hbase hbck -fixEmptyMetaCells" according to the suggestion on > > > > > http://mail-archives.apache.org/mod_mbox/hbase-dev/201312.mbox/%3CCAPQV63VrpJ-_ffOo92m=RyaZa+Z3LrpPhiNMi5Szx5uar=u...@mail.gmail.com%3E > > which > > cleared the inconsistencies and those warning messages, but the "No > > serialized HRegionInfo" and HBase inconsistencies continued to accumulate > > so some config is still not quite right. Any help is appreciated. > > > > Tom > > > >
