hello,
i am doing a lot of saves in HBase. like 100,000,000 documents, each 100KB.
before i start the program there are almost 18 connections after starting my
2 regions and one master cluster. the connections are zookeeper, hdfs and
hbase. as the process of saving starts, i have repeatedly more connections
(i guess for each time flushing) until i reach almost 80 connections and
that's when the following exeption appears. now HBase manages to save the
data somehow but it is not effective with so many connections. how do I save
this problem??
regards, shapoor
13/07/09 13:31:48 WARN client.HConnectionManager$HConnectionImplementation:
Failed all from
region=table2,doc-id-866604,1373369430484.09001c90b3d2a4c20b56c35bb976ff91.,
hostname=kcs-testhadoop02, port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
Call to kcs-testhadoop02/192.168.111.211:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.111.72:55354
remote=kcs-testhadoop02/192.168.111.211:60020]
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1598)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1450)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:916)
at at.knowcenter.backends.HBaseStorage.flush(HBaseStorage.java:324)
at at.knowcenter.evaltool.Evaluate.save(Evaluate.java:155)
at
at.knowcenter.evaltool.Evaluate.performSaveEvaluation(Evaluate.java:100)
at at.knowcenter.evaltool.Evaluate.evaluate(Evaluate.java:77)
at
at.knowcenter.evaltool.EvaluationTool.execute(EvaluationTool.java:144)
at at.knowcenter.evaltool.EvaluationTool.main(EvaluationTool.java:199)
Caused by: java.net.SocketTimeoutException: Call to
kcs-testhadoop02/192.168.111.211:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/192.168.111.72:55354 remote=kcs-testhadoop02/192.168.111.211:60020]
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1026)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:999)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at com.sun.proxy.$Proxy6.multi(Unknown Source)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1427)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1425)
at
org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:215)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1434)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1422)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.111.72:55354
remote=kcs-testhadoop02/192.168.111.211:60020]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(Unknown Source)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:373)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.DataInputStream.readInt(Unknown Source)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:646)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580)
--
View this message in context:
http://apache-hbase.679495.n3.nabble.com/60000-millis-timeout-while-waiting-for-channel-to-be-ready-for-read-tp4047612.html
Sent from the HBase Developer mailing list archive at Nabble.com.
