Hi Austin, which hbase version is in use? Is this '7b0d3c6836cf417b007771ab48d0450f' the parent region, or one of the daughter regions that got merged? What does meta scan show for the parent region and original two daughters, plus the resulting region from the merge?
Em sex, 5 de jul de 2019 às 15:25, Austin Heyne <ahe...@ccri.com> escreveu: > Hey, > > We're continuing to get some heavy use out of our normalizers (split > point pr still to come), and we're seeing an issue with daughters > getting merged. The situation is a region is splitting and becoming the > parent to regions A and B, one of these regions, say B is then getting > compacted out but region A is still waiting to be compacted. Region B is > then getting merged by our normalization plan with it's neighbor region, > C, producing region D. This leaves us with regions A and D and the > metadata for the parent. > > Now when a query comes in that spans A through D, the query planner sees > the reference to B in the parent's metadata and tries to find that > region. After the timeout period we get an exception that says the child > region (B) of the parent isn't online yet but should be soon. > > 19/07/03 18:33:06 ERROR Executor: Exception in task 3.0 in stage 9.0 > (TID 43) > org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the > location > at > > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:316) > at > > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156) > at > > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) > at > > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212) > at > org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314) > at > > org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289) > at > > org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:164) > at > org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:159) > at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:796) > > ... > Caused by: org.apache.hadoop.hbase.client.RegionOfflineException: the > only available region for the required row is a split parent, the > daughters should be online soon: > > table_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. > at > > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1307) > at > > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1183) > at > > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305) > > From what I gather we should be removing the reference to B from the > parent's metadata when B gets merged with C. > > Is this something people deal with or is there a good way to prevent this? > > Thanks, > Austin > >
