-dev (this is appropriate for user@)
Probably https://issues.apache.org/jira/browse/SPARK-10141 or
https://issues.apache.org/jira/browse/SPARK-11334 but those aren't
resolved. Feel free to jump in.
On Mon, Aug 15, 2016 at 8:13 PM, Rachana Srivastava <
rachana.srivast...@markmonitor.com> wrote:
> *Summary:*
>
> I am running Spark 1.5 on CDH5.5.1. Under extreme load intermittently I
> am getting this connection failure exception and later negative executor in
> the Spark UI.
>
>
>
> *Exception:*
>
> TRACE: org.apache.hadoop.hbase.ipc.AbstractRpcClient - Call: Multi,
> callTime: 76ms
>
> INFO : org.apache.spark.network.client.TransportClientFactory - Found
> inactive connection to /xxx.xxx.xxx., creating a new one.
>
> ERROR: org.apache.spark.network.shuffle.RetryingBlockFetcher - Exception
> while beginning fetch of 1 outstanding blocks (after 1 retries)
>
> java.io.IOException: Failed to connect to /xxx.xxx.xxx.
>
> at org.apache.spark.network.client.TransportClientFactory.
> createClient(TransportClientFactory.java:193)
>
> at org.apache.spark.network.client.TransportClientFactory.
> createClient(TransportClientFactory.java:156)
>
> at org.apache.spark.network.netty.
> NettyBlockTransferService$$anon$1.createAndStart(
> NettyBlockTransferService.scala:88)
>
> at org.apache.spark.network.shuffle.RetryingBlockFetcher.
> fetchAllOutstanding(RetryingBlockFetcher.java:140)
>
> at org.apache.spark.network.shuffle.RetryingBlockFetcher.
> access$200(RetryingBlockFetcher.java:43)
>
> at org.apache.spark.network.shuffle.RetryingBlockFetcher$
> 1.run(RetryingBlockFetcher.java:170)
>
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.
> java:262)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.net.ConnectException: Connection refused:
> /xxx.xxx.xxx.
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
> Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(
> SocketChannelImpl.java:739)
>
> at io.netty.channel.socket.nio.NioSocketChannel.
> doFinishConnect(NioSocketChannel.java:224)
>
> at io.netty.channel.nio.AbstractNioChannel$
> AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
>
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
> NioEventLoop.java:528)
>
> at io.netty.channel.nio.NioEventLoop.
> processSelectedKeysOptimized(NioEventLoop.java:468)
>
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
> NioEventLoop.java:382)
>
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.
> java:354)
>
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:111)
>
> ... 1 more
>
>
>
>
>
> *Related Defects*:
>
> https://issues.apache.org/jira/browse/SPARK-2319
>
> https://issues.apache.org/jira/browse/SPARK-9591
>
>
>
>
>
>
