[ https://issues.apache.org/jira/browse/HBASE-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852616#comment-15852616 ]
Hudson commented on HBASE-17572: -------------------------------- SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #91 (See [https://builds.apache.org/job/HBase-1.3-JDK7/91/]) HBASE-17572 HMaster: Caught throwable while processing event (apurtell: rev 054a0bd9be9e646971a5036ac9ca14bfc05f545e) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java > HMaster: Caught throwable while processing event C_M_MERGE_REGION > (UndeclaredThrowableException) > ------------------------------------------------------------------------------------------------ > > Key: HBASE-17572 > URL: https://issues.apache.org/jira/browse/HBASE-17572 > Project: HBase > Issue Type: Bug > Affects Versions: 1.3.0 > Reporter: Andrew Purtell > Assignee: Andrew Purtell > Fix For: 1.4.0, 1.3.1 > > Attachments: HBASE-17572-branch-1.3.patch > > > Running ITBLL 1B rows against branch-1.3 compiled against Hadoop 2.7.3 with > the noKill monkey policy, I see both masters go down with > master.HMaster: Caught throwable while processing event C_M_MERGE_REGION > java.lang.reflect.UndeclaredThrowableException > In ServerManager#sendRegionsMerge we call ProtobufUtil#mergeRegions, which > does a doAs, and the code within that block invokes > RSRpcServices#mergeRegions, but is not resilient against > RegionOpeningException ("region is opening") > An UndeclaredThrowableException is "thrown by a method invocation on a proxy > instance if its invocation handler's invoke method throws a checked exception > (a Throwable that is not assignable to RuntimeException or Error) that is not > assignable to any of the exception types declared in the throws clause of the > method that was invoked on the proxy instance and dispatched to the > invocation handler." > (http://docs.oracle.com/javase/7/docs/api/java/lang/reflect/UndeclaredThrowableException.html) > > {noformat} > 2017-01-31 07:21:17,495 FATAL [MASTER_TABLE_OPERATIONS-node-1:16000-0] > master.HMaster: Caught throwable while processing event C_M_MERGE_REGION > java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeRegions(ProtobufUtil.java:1990) > at > org.apache.hadoop.hbase.master.ServerManager.sendRegionsMerge(ServerManager.java:925) > at > org.apache.hadoop.hbase.master.handler.DispatchMergingRegionHandler.process(DispatchMergingRegionHandler.java:153) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: com.google.protobuf.ServiceException: > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException): > org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region > IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7. > is opening on node-2.cluster,16020,1485822382322 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:244) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:340) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.mergeRegions(AdminProtos.java:23695) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1993) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1990) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719) > ... 7 more > Caused by: > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException): > org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region > IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7. > is opening on node-2.cluster,16020,1485822382322 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) > at > org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1286) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:231) > ... 14 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)