The pasted log was from master, right ? Can you take a look at region server involved ?
Cheers On Fri, Jul 22, 2016 at 9:23 AM, Ma, Sheng-Chen (Aven) < shengchen...@esgyn.cn> wrote: > Hi Ted: > Following is part of log. I have changed log level to TRACE. But still > can’t find any clue. > > 2016-07-22 12:03:38,261 DEBUG [ProcedureExecutor-1] > procedure2.ProcedureExecutor: Procedure completed in 1.6310sec: > DisableTableProcedure (table=testTbl) id=109 owner=shengchen.ma state=F > INISHED > 2016-07-22 12:03:38,866 TRACE > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer: > RequestHeader call_id: 14 method_name: "IsMasterRunning" request_param: > true totalRequestSize: 23 bytes > 2016-07-22 12:03:38,866 TRACE > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId: > 14 service: MasterService methodName: IsMasterRunning size: 23 connection: > 192.168.3.10:49752 executing as shengchen.ma > 2016-07-22 12:03:38,866 TRACE > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId: > 14 service: MasterService methodName: IsMasterRunning size: 23 connection: > 192.168.3.10:49752 param: TODO: class > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunningRequest > connection: 192.168.3.10:49752, response is_master_running: true > queueTime: 0 processingTime: 0 totalTime: 0 > 2016-07-22 12:03:38,878 TRACE > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer: > RequestHeader call_id: 15 method_name: "getProcedureResult" request_param: > true totalRequestSize: 28 bytes > 2016-07-22 12:03:38,878 TRACE > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer: callId: 15 > service: MasterService methodName: getProcedureResult size: 28 connection: > 192.168.3.10:49752 executing as shengchen.ma > 2016-07-22 12:03:38,878 DEBUG > [B.defaultRpcServer.handler=7,queue=1,port=39479] master.MasterRpcServices: > Checking to see if procedure is done procId=109 > 2016-07-22 12:03:38,878 TRACE > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer: callId: 15 > service: MasterService methodName: getProcedureResult size: 28 connection: > 192.168.3.10:49752 param: TODO: class > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetProcedureResultRequest > connection: 192.168.3.10:49752, response state: FINISHED start_time: > 1469189016526 last_update: 1469189018157 queueTime: 0 processingTime: 0 > totalTime: 0 > 2016-07-22 12:03:38,888 TRACE > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer: > RequestHeader call_id: 16 method_name: "IsMasterRunning" request_param: > true totalRequestSize: 23 bytes > 2016-07-22 12:03:38,888 TRACE > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer: callId: 16 > service: MasterService methodName: IsMasterRunning size: 23 connection: > 192.168.3.10:49752 executing as shengchen.ma > 2016-07-22 12:03:38,888 TRACE > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer: callId: 16 > service: MasterService methodName: IsMasterRunning size: 23 connection: > 192.168.3.10:49752 param: TODO: class > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunningRequest > connection: 192.168.3.10:49752, response is_master_running: true > queueTime: 0 processingTime: 0 totalTime: 0 > 2016-07-22 12:03:39,007 TRACE > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer: > RequestHeader call_id: 17 method_name: "ModifyTable" request_param: true > totalRequestSize: 505 bytes > 2016-07-22 12:03:39,007 TRACE > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer: callId: > 17 service: MasterService methodName: ModifyTable size: 505 connection: > 192.168.3.10:49752 executing as shengchen.ma > 2016-07-22 12:03:39,047 DEBUG > [B.defaultRpcServer.handler=17,queue=2,port=39479] > util.CoprocessorClassLoader: Skipping exempt class > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint - > delegating directly to parent > 2016-07-22 12:03:39,048 DEBUG > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer: > B.defaultRpcServer.handler=17,queue=2,port=39479: callId: 17 service: > MasterService methodName: ModifyTable size: 505 connection: > 192.168.3.10:49752 > org.apache.hadoop.hbase.DoNotRetryIOException: Class > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint cannot > be loaded Set hbase.table.sanity.checks to false at conf or table > descriptor if you want to bypass sanity checks > at > org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1680) > at > org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1541) > at > org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2028) > at > org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1170) > at > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55680) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > 2016-07-22 12:03:39,622 DEBUG > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer: > RpcServer.listener,port=39479: Caught exception while reading:Connection > reset by peer > 2016-07-22 12:03:39,622 DEBUG > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer: > RpcServer.listener,port=39479: DISCONNECTING client 192.168.3.10:49752 > because read count=-1. Number of active connections: 3 > 2016-07-22 12:03:39,623 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570] > server.NIOServerCnxn: Exception causing close of session 0x1561182c97e0009 > due to java.io.IOException: Connection reset by peer > 2016-07-22 12:03:39,623 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570] > server.NIOServerCnxn: Closed socket connection for client / > 192.168.3.10:49751 which had sessionid 0x1561182c97e0009 > 2016-07-22 12:03:40,127 TRACE > [RpcServer.reader=1,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer: > RequestHeader call_id: 5466 method_name: "RegionServerReport" > request_param: true totalRequestSize: 5748 bytes > 2016-07-22 12:03:40,128 TRACE > [B.defaultRpcServer.handler=12,queue=0,port=39479] ipc.RpcServer: callId: > 5466 service: RegionServerStatusService methodName: RegionServerReport > size: 5.6 K connection: 192.168.0.17:49483 executing as traf > 2016-07-22 12:03:40,128 TRACE > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager: > 7e8aa2d93aa716ad2068808d938f0786, existingValue=-1, completeSequenceId=-1 > 2016-07-22 12:03:40,128 TRACE > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager: > 7e8aa2d93aa716ad2068808d938f0786, family=mt_, existingValue=-1, > completeSequenceId=-1 > 2016-07-22 12:03:40,128 TRACE > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager: > 7e8aa2d93aa716ad2068808d938f0786, family=tddlcf, existingValue=-1, > completeSequenceId=-1 > 2016-07-22 12:03:40,128 TRACE > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager: > 3052710e85ab46a686a22e23bff937e2, existingValue=-1, completeSequenceId=-1 > 2016-07-22 12:03:40,128 TRACE > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager: > 3052710e85ab46a686a22e23bff937e2, family=cpf, existingValue=-1, > completeSequenceId=-1 > > -----邮件原件----- > 发件人: Ted Yu [mailto:yuzhih...@gmail.com] > 发送时间: 2016年7月22日 19:48 > 收件人: user@hbase.apache.org > 主题: Re: how to Dynamic load of Coprocessors > > w.r.t. the DoNotRetryIOException, can you take a look at region server log > where testTbl region(s) was hosted ? > > See if there is some clue why the sanity check failed. > > Thanks > > On Fri, Jul 22, 2016 at 1:12 AM, Ma, Sheng-Chen (Aven) < > shengchen...@esgyn.cn> wrote: > > > Hi all: > > I want to dynamic add coprocessor in order to not restart hbase. > > > > Following is my code: > > Path path = new Path("/coprocessor_jars"); > > FileSystem fs = FileSystem.get(conf); > > FileStatus[] status = fs.listStatus(path); > > Path[] listedPaths = FileUtil.stat2Paths(status); > > for (Path p : listedPaths) { > > if(p.getName().contains("test.jar")){ > > hdfsPath = p; > > } > > } > > HBaseAdmin hadmin = new HBaseAdmin(conf); > > HTableDescriptor tableDesc = > > hadmin.getTableDescriptor("testTbl".getBytes()); > > > > tableDesc.addCoprocessor("org.apache.hadoop.hbase.coprocessor.transact > > ional.TestRegionEndpoint", > > hdfsPath, > > Coprocessor.PRIORITY_USER, null); > > // > > > tableDesc.removeCoprocessor("org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint"); > > for (Entry<ImmutableBytesWritable, ImmutableBytesWritable> entry > : > > tableDesc.getValues().entrySet()) { > > System.out.println(Bytes.toString(entry.getKey().get()) + > > " = " + Bytes.toString(entry.getValue().get())); > > } > > hadmin.disableTable("testTbl".getBytes()); > > hadmin.modifyTable("testTbl", tableDesc); > > hadmin.enableTable("testTbl"); > > > > the syso print : coprocessor$1 = hdfs:// > > 192.168.0.17:17400/coprocessor_jars/test.jar|org.apache.hadoop.hbase.c > > oprocessor.transactional.TestRegionEndpoint|1073741823| > > > > and the remote side return Exception: > > org.apache.hadoop.hbase.DoNotRetryIOException: > > org.apache.hadoop.hbase.DoNotRetryIOException: Class > > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint > > cannot be loaded Set hbase.table.sanity.checks to false at conf or > > table descriptor if you want to bypass sanity checks > > > > I use hbase 1.2 and the test.jar is not under hbase/lib, I just put > > the test.jar in hdfs. > > > > If I add test.jar to hbase/lib but not restart hbase, the upon code > > still throw same exception. > > If I add test.jar to hbase/lib and restart hbase, the upon code will > > exec successful. > > > > But my requirement is not restart hbase. > > > > Is there someone can give me a favor. > > > > Thanks > > > > > > >
