[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288941#comment-13288941 ]
Enis Soztutar commented on HBASE-6160: -------------------------------------- The exception: {code} 12/06/04 06:50:41 ERROR security.UserGroupInformation: PriviledgedActionException as: cause:org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. cannot be found in META. Exception in thread "main" org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. cannot be found in META. at org.apache.hadoop.hbase.client.MetaScanner$BlockingMetaScannerVisitor.processRow(MetaScanner.java:433) at org.apache.hadoop.hbase.client.MetaScanner$TableMetaScannerVisitor.processRow(MetaScanner.java:490) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:227) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:57) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:136) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:361) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:133) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:108) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:86) at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:326) at org.apache.hadoop.hbase.client.HTable.getRegionLocations(HTable.java:499) at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:452) at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:132) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979) {code} So the region in question is {code} 50a4617eead34cad335a8dfa727d177d {code} and from the logs we see that {{25d9c4ff574a37bd95bf5e5be6d618dd}} is split into {{1dc74065583c67b3916c4ed158cb53fa}} and {{50a4617eead34cad335a8dfa727d177d}} {code} ./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 04:56:02,855 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated, and report to master. Parent=TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd., new regions: TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa., TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.. Split took 4sec {code} After some time, {{50a4617eead34cad335a8dfa727d177d}} is further split into two: {code} ./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 05:41:13,488 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated, and report to master. Parent=TestLoadAndVerify_1 338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d., new regions: TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338802866393.16288 65d7fa8e9eec3a7d8073465296e., TestLoadAndVerify_1338798130970,]y\x04\x00\x00\x00\x00\x00/000047_0,1338802866393.413cafe6c61426e26254c197e8c0a6ba.. Split took 7sec {code} Further time passes, and CatalogJanitor deletes the META entry for that region: {code} ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:16,688 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Deleting region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. because daughter splits no longer hold references ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted daughters references, qualifier=splitA and qualifier=splitB, from parent TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,145 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. from META ./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:52:12,891 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Daughter regiondir does not exist: hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d {code} However, there is no log for the grantfather,25d9c4ff574a37bd95bf5e5be6d618dd, being deleted from META. In fact, META scan also confirms that the grandfather is in META, but not the father: {code} TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:regioninfo, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.', STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl', ENCODED => 25d9c4ff574a37bd95bf5e5be6d618dd, OFFLINE => true, SPLIT => true,} TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:server, timestamp=1338799025521, value=ip-10-226-65-102.ec2.internal:60020 TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:serverstartcode, timestamp=1338799025521, value=1338780117340 TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:splitA, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa.', STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENCODED => 1dc74065583c67b3916c4ed158cb53fa,} TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:splitB, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.', STARTKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl', ENCODED => 50a4617eead34cad335a8dfa727d177d,} {code} > META entries from daughters can be deleted before parent entries > ---------------------------------------------------------------- > > Key: HBASE-6160 > URL: https://issues.apache.org/jira/browse/HBASE-6160 > Project: HBase > Issue Type: Bug > Components: client, regionserver > Affects Versions: 0.92.2, 0.94.0, 0.96.0 > Reporter: Enis Soztutar > Assignee: Enis Soztutar > > HBASE-5986 fixed and issue, where the client sees the META entry for the > parent, but not the children. However, after the fix, we have seen the > following issue in tests: > Region A is split to -> B, C > Region B is split to -> D, E > After some time, META entry for B is deleted since it is not needed anymore, > but META entry for Region A stays in META (C still refers it). In this case, > the client throws RegionOfflineException for B. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira