[
https://issues.apache.org/jira/browse/FLINK-14359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951583#comment-16951583
]
Zheng Hu commented on FLINK-14359:
----------------------------------
When I'm running a testing sql query in my standalone env, I find the
JobManager will throw a IOException says HBase connection reset by peer:
{code}
at
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:270)
at
org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:918)
at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:230)
at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:106)
at
org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:222)
at
org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:199)
at
org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:188)
at
org.apache.flink.runtime.scheduler.LegacyScheduler.<init>(LegacyScheduler.java:65)
at
org.apache.flink.runtime.scheduler.LegacySchedulerFactory.createInstance(LegacySchedulerFactory.java:70)
at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278)
at
org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:266)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)
at
org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:146)
... 10 more
Caused by:
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=3, exceptions:
Tue Oct 15 12:26:27 CST 2019, RpcRetryingCaller{globalStartTime=1571113587249,
pause=100, retries=3},
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.MasterNotRunningException:
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.shaded.com.google.protobuf.ServiceException:
java.io.IOException: Call to /30.5.17.5:55263 failed on local exception:
java.io.IOException: Connection reset by peer
Tue Oct 15 12:26:27 CST 2019, RpcRetryingCaller{globalStartTime=1571113587249,
pause=100, retries=3},
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.MasterNotRunningException:
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.shaded.com.google.protobuf.ServiceException:
java.io.IOException: Call to /30.5.17.5:55263 failed on local exception:
java.io.IOException: Connection reset by peer
Tue Oct 15 12:26:27 CST 2019, RpcRetryingCaller{globalStartTime=1571113587249,
pause=100, retries=3},
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.MasterNotRunningException:
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.shaded.com.google.protobuf.ServiceException:
java.io.IOException: Call to /30.5.17.5:55263 failed on local exception:
java.io.IOException: Connection reset by peer
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:164)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4447)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4439)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:440)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:424)
at
org.apache.flink.addons.hbase.AbstractTableInputFormat.createInputSplits(AbstractTableInputFormat.java:215)
at
org.apache.flink.addons.hbase.AbstractTableInputFormat.createInputSplits(AbstractTableInputFormat.java:50)
at
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:256)
... 23 more
Caused by:
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.MasterNotRunningException:
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.shaded.com.google.protobuf.ServiceException:
java.io.IOException: Call to /30.5.17.5:55263 failed on local exception:
java.io.IOException: Connection reset by peer
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1608)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1628)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1785)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:140)
... 30 more
Caused by:
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.shaded.com.google.protobuf.ServiceException:
java.io.IOException: Call to /30.5.17.5:55263 failed on local exception:
java.io.IOException: Connection reset by peer
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:335)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:94)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1639)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1577)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1599)
... 34 more
Caused by: java.io.IOException: Call to /30.5.17.5:55263 failed on local
exception: java.io.IOException: Connection reset by peer
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.BlockingRpcConnection.closeConn(BlockingRpcConnection.java:676)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.BlockingRpcConnection.readResponse(BlockingRpcConnection.java:642)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.BlockingRpcConnection.run(BlockingRpcConnection.java:334)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.BlockingRpcConnection.readResponse(BlockingRpcConnection.java:574)
... 2 more
{code}
And after Enable the HBase debug log at server side, it says:
{code}
2019-10-15 12:26:27,491 DEBUG
[RpcServer.reader=4,bindAddress=0.0.0.0,port=55263] ipc.RpcServer:
RpcServer.listener,port=55263: Caught exception while reading:
org.apache.hadoop.hbase.ipc.UnsupportedCellCodecException:
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.codec.KeyValueCodec
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.setupCellBlockCodecs(RpcServer.java:1892)
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processConnectionHeader(RpcServer.java:1826)
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1947)
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1790)
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1769)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:955)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:725)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:701)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.codec.KeyValueCodec
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.setupCellBlockCodecs(RpcServer.java:1890)
... 10 more
{code}
So seems the HBase key value codec class has been shaded also, the server can
not find the shaded class now, let me think how to resolve this issue.
> Create a module called flink-sql-connector-hbase to shade HBase
> ---------------------------------------------------------------
>
> Key: FLINK-14359
> URL: https://issues.apache.org/jira/browse/FLINK-14359
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / HBase
> Reporter: Jingsong Lee
> Assignee: Zheng Hu
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We need do the same thing as kafka and elasticsearch to HBase.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
