my hbase replication has stopped
I am on hbase version 1.0.0-cdh5.4.8 (Cloudera build)
I have 2 clusters in 2 different datacenters
1 is master the other is slave
I see the following errors in log
2016-04-13 22:32:50,217 WARN
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint:
Can't replicate because of a local or network error:
java.io.IOException: Call to
hadoop2-private.sjc03.infra.com/10.160.22.99:60020 failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
id=1014, waitTime=1200001, operationTimeout=1200000 expired.
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1255)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1223)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:21783)
at
org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:65)
at
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:161)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:696)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:410)
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
id=1014, waitTime=1200001, operationTimeout=1200000 expired.
at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1197)
... 7 more
which in turn fills the queue and I get
2016-04-13 22:35:19,555 WARN
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint:
Can't replicate because of an error on the remote cluster:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.RpcServer$CallQueueTooBigException):
Call queue is full on /0.0.0.0:60020, is
hbase.ipc.server.max.callqueue.size too small?
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1219)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:21783)
at
org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:65)
at
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:161)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:696)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:410)
My peers look good and this was working until Mar 27
we did have an inadvertent outage but I was able to restore all cluster
services
status 'replication'
version 1.0.0-cdh5.4.8
5 live servers
hadoop5-private.wdc01.infra.com:
SOURCE: PeerID=1, AgeOfLastShippedOp=1538240180,
SizeOfLogQueue=2135, TimeStampsOfLastShippedOp=Sun Mar 27 04:00:42
GMT+00:00 2016, Replication Lag=1539342209
SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Tue Mar
22 10:09:39 GMT+00:00 2016
hadoop2-private.wdc01.infra.com:
SOURCE: PeerID=1, AgeOfLastShippedOp=810222876,
SizeOfLogQueue=1302, TimeStampsOfLastShippedOp=Mon Apr 04 14:31:37
GMT+00:00 2016, Replication Lag=810287122
SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Fri Mar
25 21:20:59 GMT+00:00 2016
hadoop4-private.wdc01.infra.com:
SOURCE: PeerID=1, AgeOfLastShippedOp=602417946,
SizeOfLogQueue=190, TimeStampsOfLastShippedOp=Thu Apr 07 00:06:38
GMT+00:00 2016, Replication Lag=602983605
SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Mon Apr
04 14:35:56 GMT+00:00 2016
hadoop1-private.wdc01.infra.com:
SOURCE: PeerID=1, AgeOfLastShippedOp=602574285,
SizeOfLogQueue=183, TimeStampsOfLastShippedOp=Thu Apr 07 00:10:29
GMT+00:00 2016, Replication Lag=602753383
SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Apr
07 00:10:23 GMT+00:00 2016
hadoop3-private.wdc01.infra.com:
SOURCE: PeerID=1, AgeOfLastShippedOp=602002192,
SizeOfLogQueue=1148, TimeStampsOfLastShippedOp=Thu Apr 07 00:06:52
GMT+00:00 2016, Replication Lag=602971172
SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Apr
07 00:06:50 GMT+00:00 2016
I can curl the quorum I set so I don't think its network
What can I do to troubleshoot?
Tried to run the following
hbase org.apache.hadoop.hbase.replication.regionserver.ReplicationSyncUp 100000
got the following response
16/04/13 23:37:17 INFO zookeeper.ClientCnxn: Socket connection
established, initiating session, client: /10.125.122.237:50784,
server: hadoop2-private.sjc03.infra.com/10.160.22.99:2181
16/04/13 23:37:17 INFO zookeeper.ClientCnxn: Session establishment
complete on server hadoop2-private.sjc03.infra.com/10.160.22.99:2181,
sessionid = 0x252f1a90269f5d6, negotiated timeout = 150000
16/04/13 23:37:17 INFO regionserver.ReplicationSource: Replicating
de6643f5-2a36-413e-b55f-8840b26395b1 ->
06a68811-0e50-4802-a478-d199df96bf85
16/04/13 23:37:27 INFO regionserver.ReplicationSource: Closing source
1 because: Region server is closing
16/04/13 23:37:27 WARN regionserver.ReplicationSource: Interrupted
while reading edits
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
at
java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:553)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.getNextPath(ReplicationSource.java:489)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:308)
16/04/13 23:37:27 INFO zookeeper.ZooKeeper: Session: 0x252f1a90269f5d6 closed
16/04/13 23:37:27 INFO zookeeper.ClientCnxn: EventThread shut down
16/04/13 23:37:27 INFO
client.ConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x152f1a8ff4ef600
16/04/13 23:37:27 INFO zookeeper.ZooKeeper: Session: 0x152f1a8ff4ef600 closed
16/04/13 23:37:27 INFO zookeeper.ClientCnxn: EventThread shut down
16/04/13 23:37:31 INFO zookeeper.ZooKeeper: Session: 0x153ee0d274c3c6a closed
16/04/13 23:37:31 INFO zookeeper.ClientCnxn: EventThread shut down
I am willing to lose the queue if there is a way to flush it and reset the
sync process, because I can distscp of various data and manually load my
tables to play catchup
Or if there are other things I should try to diagnose to find where the log
jam is
