[
https://issues.apache.org/jira/browse/HBASE-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reopened HBASE-17572:
------------------------------------
Reopening.
I was able to reproduce again with branch-1 (not branch-1.3). I misjudged this.
In retrospect it's obvious, the exception thrown out of the doAs is
UndeclaredThrowableException. We have to catch transform the ServiceException
inside the doAs block. Will revert the earlier commit and replace with the
proper fix.
> HMaster: Caught throwable while processing event C_M_MERGE_REGION
> (UndeclaredThrowableException)
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-17572
> URL: https://issues.apache.org/jira/browse/HBASE-17572
> Project: HBase
> Issue Type: Bug
> Affects Versions: 1.3.0
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 1.4.0, 1.3.1
>
> Attachments: HBASE-17572-branch-1.3.patch
>
>
> Running ITBLL 1B rows against branch-1.3 compiled against Hadoop 2.7.3 with
> the noKill monkey policy, I see both masters go down with
> master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
> java.lang.reflect.UndeclaredThrowableException
> In ServerManager#sendRegionsMerge we call ProtobufUtil#mergeRegions, which
> does a doAs, and the code within that block invokes
> RSRpcServices#mergeRegions, but is not resilient against
> RegionOpeningException ("region is opening")
> An UndeclaredThrowableException is "thrown by a method invocation on a proxy
> instance if its invocation handler's invoke method throws a checked exception
> (a Throwable that is not assignable to RuntimeException or Error) that is not
> assignable to any of the exception types declared in the throws clause of the
> method that was invoked on the proxy instance and dispatched to the
> invocation handler."
> (http://docs.oracle.com/javase/7/docs/api/java/lang/reflect/UndeclaredThrowableException.html)
>
> {noformat}
> 2017-01-31 07:21:17,495 FATAL [MASTER_TABLE_OPERATIONS-node-1:16000-0]
> master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeRegions(ProtobufUtil.java:1990)
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionsMerge(ServerManager.java:925)
> at
> org.apache.hadoop.hbase.master.handler.DispatchMergingRegionHandler.process(DispatchMergingRegionHandler.java:153)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException):
> org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
> IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7.
> is opening on node-2.cluster,16020,1485822382322
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:244)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:340)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.mergeRegions(AdminProtos.java:23695)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1993)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1990)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719)
> ... 7 more
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException):
> org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
> IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7.
> is opening on node-2.cluster,16020,1485822382322
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1286)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:231)
> ... 14 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
