On Mon, Apr 20, 2020 at 7:59 AM Mike Linsinbigler <mlinsinbig...@ccri.com> wrote:
> Hello, > > I am running HBase 1.4.10 on AWS EMR 5.29.0 and have had issues after > some of our Region Servers restart (due to a variety of reasons) and the > cluster enters an inconsistent state after they come back up. > > Running hbck, I am presented with many instances of: > > ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta > > > I am able to resolve this issue by running: > > "hbase hbck -fixEmptyMetaCells" > > > However, this is only a fix until the next time one of our region > servers restart which is currently a daily event. Does anyone know how > to prevent this issue from occurring in the first place? It looks like > our Region Server was in the middle of splitting and compaction > operations before aborting. > > I've noticed that writing to hbase while these errors are present can > result in ingest issues within our application so I'd really like to > understand how the meta table can get into this state. > > Rows in hbase:meta w/ empty REGIONINFO_QUALIFIER will mess you up. Shoudn't be happening. Have you tried tracing the lifecycle of one of these empty rows? The row name is the region name. Search master logs over time using the regionname. See if you can get a sense of what is happening manufacturing empty rows. Yours, S > Thanks, > > Mike > >
