[
https://issues.apache.org/jira/browse/HBASE-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720911#comment-13720911
]
stack commented on HBASE-8764:
------------------------------
Here is commit message for this patch updated:
{code}
Add retrying of Master operations; helps when running hbase-it and chaos monkey
kills
Master or Master is not yet up ready to take on operations.
Refactors ServerCallable. ServerCallable had a public call() method and then
beside
it a withRetries() and also a withoutRetries(). Confusing. Also the rpc
retrying
with its specific handling of server exception returns was not reusable buried
down
in ServerCallable guts.
This patch moves the rpc retrying code out of ServerCallable into a utility
RpcRetryingCaller class (A 'Caller' runs the 'Callable'). ServerCallable
shrinks,
implements a new RetryingCallable Interface, and becomes RegionServerCallable,
a class
that is just about Calling -- no rpc nor retries, a Callable class with added
details
on where the Callable is to be applied (table name and row), -- etc.
This pattern is then applied to Master operations. Master operations were not
retried
previously. The Master operation Callables are now like RegionServerCallable
(though
they need to carry way less detail), implement RetryingCallable, and are passed
to
RpcRetryingCaller so they are retried. Changed some exceptions so they now
implement
DoNotRetryException because not all master operations should be retried.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
Move to new style making calls (create a Callable and a Caller to run the op)
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
Create a caller on initialization and reuse it doing calls in new style
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
Javadoc.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
Use new style making calls where we first create the Callable and then a
Caller of the right type to make the call. Remove the execute methods.
Have all directly use executeCallable instead. HBaseAdmin manipulates two
'admin' rpc Services -- the 'Monitor' service and the 'Admin' service.
Amend the Callables that were declared as inner classes here so they both
share a base class -- MasterCallable -- that does shared functionality
and that implements RetryingCallable. Also adds support for closing the
service needed by the backing AsyncProcess.
M
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
M
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java
Remove deprecated methods.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
Use new model calling.
M
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
Implement RegionServerCallable instead of ServerCallable.
A
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java
ServerCallable but stripped of retrying.
M
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedException.java
Javadoc
A
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetryingCallable.java
Interface implemented by Callables that we can retry.
A
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java
This is the retrying functionality broken out of what was ServerCallable.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
Implement RegionServerCallable.
A hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCaller.java
A subclass of RpcRetryingCaller that supplies the type returned when scanning.
D hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java
Replaced by RpcRetryingCaller and RegionServerCallable.
M
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/DoNotRetryIOException.java
Comment
M
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/HBaseSnapshotException.java
M
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/InvalidFamilyOperationException.java
M
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/TableExistsException.java
M
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/LockTimeoutException.java
M
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/TableNotDisabledException.java
M
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/TableNotEnabledException.java
M
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/TableNotFoundException.java
Implement DoNotRetryException
M
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionCoprocessorRpcChannel.java
M
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
M
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
M
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
M
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
New model.
M hbase-common/src/main/java/org/apache/hadoop/hbase/util/Threads.java
Javadoc. Perpeturate interrupt.
M
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
Javadoc.
A
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHBaseAdminNoCluster.java
Unit test.
Converted the rest of the tests to new model
{code}
> Some MasterMonitorCallable should retry
> ---------------------------------------
>
> Key: HBASE-8764
> URL: https://issues.apache.org/jira/browse/HBASE-8764
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
> Affects Versions: 0.95.1
> Reporter: Elliott Clark
> Assignee: stack
> Fix For: 0.95.2
>
> Attachments: 8764.txt, 8764v2.txt, 8764v3.txt, 8764v4.txt,
> 8796v5.txt, 8796v7.txt
>
>
> Calls in the admin that only get status should re-try.
> got a call stack like:
> {code}
> org.apache.hadoop.hbase.exceptions.PleaseHoldException:
> org.apache.hadoop.hbase.exceptions.PleaseHoldException: Master is initializing
> at
> org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2266)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1610)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1646)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:20930)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:230)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2705)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.execute(HBaseAdmin.java:2674)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:524)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:417)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:349)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.createSchema(IntegrationTestBigLinkedList.java:437)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.runGenerator(IntegrationTestBigLinkedList.java:471)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.run(IntegrationTestBigLinkedList.java:505)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.runGenerator(IntegrationTestBigLinkedList.java:698)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.run(IntegrationTestBigLinkedList.java:748)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedListWithChaosMonkey.testContinuousIngest(IntegrationTestBigLinkedListWithChaosMonkey.java:80)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:117)
> at
> org.apache.hadoop.hbase.IntegrationTestsDriver.doWork(IntegrationTestsDriver.java:111)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:108)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.IntegrationTestsDriver.main(IntegrationTestsDriver.java:47)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
