The pastebin of master log snippet doesn't provide enough clue. Can you trace, say, 1042331bd51614fabf01034113804665, back in master log to see whether it went through splits ? Please pastebin related portion of master log.
Cheers On Fri, May 29, 2015 at 12:29 PM, Tom Chan <tc...@prediction.io> wrote: > On Thu, May 28, 2015 at 7:53 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > What release of hbase are you using ? > > > I'm using 0.98.9. I should have mentioned it before anything else. > > > > > If you trace 5fe111d3302cefb2e96d541a489ff026 in master log, do you find > > some clue ? > > > From the timestamp of the logs, all these warnings are given out when I do > "pio status" which tests if hbase is up by writing to it a table > "predictionio_eventdata:events_0" then deleting it. > > > > > Pastebin related log snippet if possible. > > > > http://pastebin.com/WDV01WQ7. There is only one line of "WARN No > HRegionInfo" instead of thousands of lines since I issued the "hbase hbck > -fixEmptyMetaCells" just now (but it keeps accumulating and that causes > problem). > > Also one thing I notice is that before fixing the meta table > (-fixEmptyMetaCells), in hbase shell, counting the non-existing table > 'predictionio_eventdata:events_0' takes much longer (a minute or two) than > some other non-existing tables. It spits out this line (or more) before > prompting that the table does not exist: > > ERROR: HRegionInfo was null or empty in hbase:meta, > > row=keyvalues={predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:seqnumDuringOpen/1432926325684/Put/vlen=8/mvcc=0, > > predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:server/1432926325684/Put/vlen=46/mvcc=0, > > predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:serverstartcode/1432926325684/Put/vlen=8/mvcc=0} > > Tom > > > > > > Thanks > > > > On Thu, May 28, 2015 at 6:48 PM, Tom Chan <tc...@prediction.io> wrote: > > > > > Hi, > > > > > > I kept running into this whenever I remove tables > > > > > > WARN catalog.MetaReader: No serialized HRegionInfo in > > > > > > > > > keyvalues={predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:seqnumDuringOpen/1430338711901/Put/vlen=8/mvcc=0, > > > > > > > > > predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:server/1430338711901/Put/vlen=46/mvcc=0, > > > > > > > > > predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:serverstartcode/1430338711901/Put/vlen=8/mvcc=0} > > > > > > create tables > > > > > > WARN client.HBaseAdmin: No serialized HRegionInfo in > > > > > > > > > keyvalues={predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:seqnumDuringOpen/1427845220983/Put/vlen=8/mvcc=0, > > > > > > > > > predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:server/1427845220983/Put/vlen=46/mvcc=0, > > > > > > > > > predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:serverstartcode/1427845220983/Put/vlen=8/mvcc=0} > > > > > > > > > When doing hbck, I found some 1800 inconsistencies of the form > > > > > > ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta > > > > > > > > > > > > keyvalues={predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:seqnumDuringOpen/1431758974883/Put/vlen=8/mvcc=0, > > > > > > > > > predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:server/1431758974883/Put/vlen=46/mvcc=0, > > > > > > > > > predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:serverstartcode/1431758974883/Put/vlen=8/mvcc=0} > > > > > > while all the tables are okay. > > > > > > I ran "hbase hbck -fixEmptyMetaCells" according to the suggestion on > > > > > > > > > http://mail-archives.apache.org/mod_mbox/hbase-dev/201312.mbox/%3CCAPQV63VrpJ-_ffOo92m=RyaZa+Z3LrpPhiNMi5Szx5uar=u...@mail.gmail.com%3E > > > which > > > cleared the inconsistencies and those warning messages, but the "No > > > serialized HRegionInfo" and HBase inconsistencies continued to > accumulate > > > so some config is still not quite right. Any help is appreciated. > > > > > > Tom > > > > > > > >
