This is definitely not expected behavior (either on S3 or HDFS). I agree with Stack's request to try to trace how these are happening. Since this is with EMR, you can also create a customer support ticket to help debug the issue.
On Wed, Apr 22, 2020, 1:58 AM Wellington Chevreuil < [email protected]> wrote: > > > > I am running HBase 1.4.10 on AWS EMR 5.29.0 > > > Which file system is hbase root dir under? If it's S3, it's very likely > some of its lack of consistency issues leading to missing regioninfo files. > > Em qua., 22 de abr. de 2020 às 02:21, Stack <[email protected]> escreveu: > > > On Mon, Apr 20, 2020 at 7:59 AM Mike Linsinbigler < > [email protected]> > > wrote: > > > > > Hello, > > > > > > I am running HBase 1.4.10 on AWS EMR 5.29.0 and have had issues after > > > some of our Region Servers restart (due to a variety of reasons) and > the > > > cluster enters an inconsistent state after they come back up. > > > > > > Running hbck, I am presented with many instances of: > > > > > > ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta > > > > > > > > > I am able to resolve this issue by running: > > > > > > "hbase hbck -fixEmptyMetaCells" > > > > > > > > > However, this is only a fix until the next time one of our region > > > servers restart which is currently a daily event. Does anyone know how > > > to prevent this issue from occurring in the first place? It looks like > > > our Region Server was in the middle of splitting and compaction > > > operations before aborting. > > > > > > I've noticed that writing to hbase while these errors are present can > > > result in ingest issues within our application so I'd really like to > > > understand how the meta table can get into this state. > > > > > > > > Rows in hbase:meta w/ empty REGIONINFO_QUALIFIER will mess you up. > Shoudn't > > be happening. Have you tried tracing the lifecycle of one of these empty > > rows? The row name is the region name. Search master logs over time using > > the regionname. See if you can get a sense of what is happening > > manufacturing empty rows. > > > > Yours, > > S > > > > > > > > > Thanks, > > > > > > Mike > > > > > > > > >
