[ https://issues.apache.org/jira/browse/HBASE-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-17572: ----------------------------------- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.3.1 1.4.0 Status: Resolved (was: Patch Available) I make a quick pass over all of ProtobufUtil and the other handlers and this is the only one affected. Change only applies to branch-1.3 and branch-1. branch-1.2 and below don't have the doAs addition where the ServiceException could bubble out. Master has a different handler implementation. > HMaster: Caught throwable while processing event C_M_MERGE_REGION > (UndeclaredThrowableException) > ------------------------------------------------------------------------------------------------ > > Key: HBASE-17572 > URL: https://issues.apache.org/jira/browse/HBASE-17572 > Project: HBase > Issue Type: Bug > Affects Versions: 1.3.0 > Reporter: Andrew Purtell > Assignee: Andrew Purtell > Fix For: 1.4.0, 1.3.1 > > Attachments: HBASE-17572-branch-1.3.patch > > > Running ITBLL 1B rows against branch-1.3 compiled against Hadoop 2.7.3 with > the noKill monkey policy, I see both masters go down with > master.HMaster: Caught throwable while processing event C_M_MERGE_REGION > java.lang.reflect.UndeclaredThrowableException > In ServerManager#sendRegionsMerge we call ProtobufUtil#mergeRegions, which > does a doAs, and the code within that block invokes > RSRpcServices#mergeRegions, but is not resilient against > RegionOpeningException ("region is opening") > An UndeclaredThrowableException is "thrown by a method invocation on a proxy > instance if its invocation handler's invoke method throws a checked exception > (a Throwable that is not assignable to RuntimeException or Error) that is not > assignable to any of the exception types declared in the throws clause of the > method that was invoked on the proxy instance and dispatched to the > invocation handler." > (http://docs.oracle.com/javase/7/docs/api/java/lang/reflect/UndeclaredThrowableException.html) > > {noformat} > 2017-01-31 07:21:17,495 FATAL [MASTER_TABLE_OPERATIONS-node-1:16000-0] > master.HMaster: Caught throwable while processing event C_M_MERGE_REGION > java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeRegions(ProtobufUtil.java:1990) > at > org.apache.hadoop.hbase.master.ServerManager.sendRegionsMerge(ServerManager.java:925) > at > org.apache.hadoop.hbase.master.handler.DispatchMergingRegionHandler.process(DispatchMergingRegionHandler.java:153) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: com.google.protobuf.ServiceException: > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException): > org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region > IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7. > is opening on node-2.cluster,16020,1485822382322 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:244) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:340) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.mergeRegions(AdminProtos.java:23695) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1993) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1990) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719) > ... 7 more > Caused by: > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException): > org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region > IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7. > is opening on node-2.cluster,16020,1485822382322 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) > at > org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1286) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:231) > ... 14 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)