Hi Vikas,
Are you running this on your local laptop etc or using some IDE etc?
What is your available memory for Spark?
Start with minimum set like below
def spark_session_local(appName):
return SparkSession.builder \
.master('local[1]') \
.appName(appName) \
.enableHiveSupport() \
.getOrCreate()
HTH
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Wed, 20 Jan 2021 at 12:32, Vikas Garg <sperry...@gmail.com> wrote:
> Hi Sachit,
>
> I am running it in local. My IP mentioned is the private IP address and
> therefore it is useless for anyone.
>
> On Wed, 20 Jan 2021 at 17:37, Sachit Murarka <connectsac...@gmail.com>
> wrote:
>
>> Hi Vikas
>>
>> 1. Are you running in local mode? Master has local[*]
>> 2. Pls mask the ip or confidential info while sharing logs
>>
>> Thanks
>> Sachit
>>
>> On Wed, 20 Jan 2021, 17:35 Vikas Garg, <sperry...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am facing issue with spark executor. I am struggling with this issue
>>> since last many days and unable to resolve the issue.
>>>
>>> Below is the configuration I have given.
>>>
>>> val spark = SparkSession.builder()
>>> .appName("Spark Job")
>>> .master("local[*]")
>>> .config("spark.dynamicAllocation.enabled", true)
>>> .config("spark.shuffle.service.enabled", true)
>>> .config("spark.driver.maxResultSize", "8g")
>>> .config("spark.driver.memory", "8g")
>>> .config("spark.executor.memory", "8g")
>>> .config("spark.network.timeout", "3600s")
>>> .getOrCreate()
>>>
>>> 1/01/20 17:06:57 ERROR RetryingBlockFetcher: Exception while beginning
>>> fetch of 1 outstanding blocks
>>>
>>> *java.io.IOException*: Failed to connect to
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:253*)
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:195*)
>>>
>>> at
>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(
>>> *NettyBlockTransferService.scala:122*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(
>>> *RetryingBlockFetcher.java:141*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.start(
>>> *RetryingBlockFetcher.java:121*)
>>>
>>> at
>>> org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks(
>>> *NettyBlockTransferService.scala:143*)
>>>
>>> at
>>> org.apache.spark.network.BlockTransferService.fetchBlockSync(
>>> *BlockTransferService.scala:103*)
>>>
>>> at
>>> org.apache.spark.storage.BlockManager.fetchRemoteManagedBuffer(
>>> *BlockManager.scala:1010*)
>>>
>>> at
>>> org.apache.spark.storage.BlockManager.$anonfun$getRemoteBlock$8(
>>> *BlockManager.scala:954*)
>>>
>>> at scala.Option.orElse(*Option.scala:289*)
>>>
>>> at org.apache.spark.storage.BlockManager.getRemoteBlock(
>>> *BlockManager.scala:954*)
>>>
>>> at org.apache.spark.storage.BlockManager.getRemoteBytes(
>>> *BlockManager.scala:1092*)
>>>
>>> at
>>> org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1(
>>> *TaskResultGetter.scala:88*)
>>>
>>> at
>>> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
>>>
>>> at org.apache.spark.util.Utils$.logUncaughtExceptions(
>>> *Utils.scala:1932*)
>>>
>>> at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(
>>> *TaskResultGetter.scala:63*)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> *ThreadPoolExecutor.java:1149*)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> *ThreadPoolExecutor.java:624*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*:
>>> Permission denied: no further information:
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> Caused by: *java.net.SocketException*: Permission denied: no further
>>> information
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*
>>> )
>>>
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(
>>> *SocketChannelImpl.java:715*)
>>>
>>> at
>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(
>>> *NioSocketChannel.java:330*)
>>>
>>> at
>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(
>>> *AbstractNioChannel.java:334*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
>>> *NioEventLoop.java:702*)
>>>
>>> at
>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
>>> *NioEventLoop.java:650*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
>>> *NioEventLoop.java:576*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.run(
>>> *NioEventLoop.java:493*)
>>>
>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(
>>> *SingleThreadEventExecutor.java:989*)
>>>
>>> at io.netty.util.internal.ThreadExecutorMap$2.run(
>>> *ThreadExecutorMap.java:74*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> 21/01/20 17:06:57 ERROR RetryingBlockFetcher: Exception while beginning
>>> fetch of 1 outstanding blocks
>>>
>>> *java.io.IOException*: Failed to connect to
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:253*)
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:195*)
>>>
>>> at
>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(
>>> *NettyBlockTransferService.scala:122*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(
>>> *RetryingBlockFetcher.java:141*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.start(
>>> *RetryingBlockFetcher.java:121*)
>>>
>>> at
>>> org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks(
>>> *NettyBlockTransferService.scala:143*)
>>>
>>> at
>>> org.apache.spark.network.BlockTransferService.fetchBlockSync(
>>> *BlockTransferService.scala:103*)
>>>
>>> at
>>> org.apache.spark.storage.BlockManager.fetchRemoteManagedBuffer(
>>> *BlockManager.scala:1010*)
>>>
>>> at
>>> org.apache.spark.storage.BlockManager.$anonfun$getRemoteBlock$8(
>>> *BlockManager.scala:954*)
>>>
>>> at scala.Option.orElse(*Option.scala:289*)
>>>
>>> at org.apache.spark.storage.BlockManager.getRemoteBlock(
>>> *BlockManager.scala:954*)
>>>
>>> at org.apache.spark.storage.BlockManager.getRemoteBytes(
>>> *BlockManager.scala:1092*)
>>>
>>> at
>>> org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1(
>>> *TaskResultGetter.scala:88*)
>>>
>>> at
>>> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
>>>
>>> at org.apache.spark.util.Utils$.logUncaughtExceptions(
>>> *Utils.scala:1932*)
>>>
>>> at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(
>>> *TaskResultGetter.scala:63*)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> *ThreadPoolExecutor.java:1149*)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> *ThreadPoolExecutor.java:624*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*:
>>> Permission denied: no further information:
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> Caused by: *java.net.SocketException*: Permission denied: no further
>>> information
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*
>>> )
>>>
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(
>>> *SocketChannelImpl.java:715*)
>>>
>>> at
>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(
>>> *NioSocketChannel.java:330*)
>>>
>>> at
>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(
>>> *AbstractNioChannel.java:334*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
>>> *NioEventLoop.java:702*)
>>>
>>> at
>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
>>> *NioEventLoop.java:650*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
>>> *NioEventLoop.java:576*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.run(
>>> *NioEventLoop.java:493*)
>>>
>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(
>>> *SingleThreadEventExecutor.java:989*)
>>>
>>> at io.netty.util.internal.ThreadExecutorMap$2.run(
>>> *ThreadExecutorMap.java:74*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> 21/01/20 17:07:02 ERROR RetryingBlockFetcher: Exception while beginning
>>> fetch of 1 outstanding blocks (after 1 retries)
>>>
>>> *java.io.IOException*: Failed to connect to
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:253*)
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:195*)
>>>
>>> at
>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(
>>> *NettyBlockTransferService.scala:122*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(
>>> *RetryingBlockFetcher.java:141*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0(
>>> *RetryingBlockFetcher.java:169*)
>>>
>>> at java.util.concurrent.Executors$RunnableAdapter.call(
>>> *Executors.java:511*)
>>>
>>> at java.util.concurrent.FutureTask.run(*FutureTask.java:266*
>>> )
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> *ThreadPoolExecutor.java:1149*)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> *ThreadPoolExecutor.java:624*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*:
>>> Permission denied: no further information:
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> Caused by: *java.net.SocketException*: Permission denied: no further
>>> information
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*
>>> )
>>>
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(
>>> *SocketChannelImpl.java:715*)
>>>
>>> at
>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(
>>> *NioSocketChannel.java:330*)
>>>
>>> at
>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(
>>> *AbstractNioChannel.java:334*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
>>> *NioEventLoop.java:702*)
>>>
>>> at
>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
>>> *NioEventLoop.java:650*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
>>> *NioEventLoop.java:576*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.run(
>>> *NioEventLoop.java:493*)
>>>
>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(
>>> *SingleThreadEventExecutor.java:989*)
>>>
>>> at io.netty.util.internal.ThreadExecutorMap$2.run(
>>> *ThreadExecutorMap.java:74*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> 21/01/20 17:07:02 ERROR RetryingBlockFetcher: Exception while beginning
>>> fetch of 1 outstanding blocks (after 1 retries)
>>>
>>> *java.io.IOException*: Failed to connect to
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:253*)
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:195*)
>>>
>>> at
>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(
>>> *NettyBlockTransferService.scala:122*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(
>>> *RetryingBlockFetcher.java:141*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0(
>>> *RetryingBlockFetcher.java:169*)
>>>
>>> at java.util.concurrent.Executors$RunnableAdapter.call(
>>> *Executors.java:511*)
>>>
>>> at java.util.concurrent.FutureTask.run(*FutureTask.java:266*
>>> )
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> *ThreadPoolExecutor.java:1149*)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> *ThreadPoolExecutor.java:624*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*:
>>> Permission denied: no further information:
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> Caused by: *java.net.SocketException*: Permission denied: no further
>>> information
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*
>>> )
>>>
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(
>>> *SocketChannelImpl.java:715*)
>>>
>>> at
>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(
>>> *NioSocketChannel.java:330*)
>>>
>>> at
>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(
>>> *AbstractNioChannel.java:334*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
>>> *NioEventLoop.java:702*)
>>>
>>> at
>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
>>> *NioEventLoop.java:650*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
>>> *NioEventLoop.java:576*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.run(
>>> *NioEventLoop.java:493*)
>>>
>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(
>>> *SingleThreadEventExecutor.java:989*)
>>>
>>> at io.netty.util.internal.ThreadExecutorMap$2.run(
>>> *ThreadExecutorMap.java:74*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> 21/01/20 17:07:07 ERROR RetryingBlockFetcher: Exception while beginning
>>> fetch of 1 outstanding blocks (after 2 retries)
>>>
>>> *java.io.IOException*: Failed to connect to
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:253*)
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:195*)
>>>
>>> at
>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(
>>> *NettyBlockTransferService.scala:122*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(
>>> *RetryingBlockFetcher.java:141*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0(
>>> *RetryingBlockFetcher.java:169*)
>>>
>>> at java.util.concurrent.Executors$RunnableAdapter.call(
>>> *Executors.java:511*)
>>>
>>> at java.util.concurrent.FutureTask.run(*FutureTask.java:266*
>>> )
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> *ThreadPoolExecutor.java:1149*)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> *ThreadPoolExecutor.java:624*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*:
>>> Permission denied: no further information:
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> Caused by: *java.net.SocketException*: Permission denied: no further
>>> information
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*
>>> )
>>>
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(
>>> *SocketChannelImpl.java:715*)
>>>
>>> at
>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(
>>> *NioSocketChannel.java:330*)
>>>
>>> at
>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(
>>> *AbstractNioChannel.java:334*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
>>> *NioEventLoop.java:702*)
>>>
>>> at
>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
>>> *NioEventLoop.java:650*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
>>> *NioEventLoop.java:576*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.run(
>>> *NioEventLoop.java:493*)
>>>
>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(
>>> *SingleThreadEventExecutor.java:989*)
>>>
>>> at io.netty.util.internal.ThreadExecutorMap$2.run(
>>> *ThreadExecutorMap.java:74*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> 21/01/20 17:07:07 ERROR RetryingBlockFetcher: Exception while beginning
>>> fetch of 1 outstanding blocks (after 2 retries)
>>>
>>> *java.io.IOException*: Failed to connect to
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:253*)
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:195*)
>>>
>>> at
>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(
>>> *NettyBlockTransferService.scala:122*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(
>>> *RetryingBlockFetcher.java:141*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0(
>>> *RetryingBlockFetcher.java:169*)
>>>
>>> at java.util.concurrent.Executors$RunnableAdapter.call(
>>> *Executors.java:511*)
>>>
>>> at java.util.concurrent.FutureTask.run(*FutureTask.java:266*
>>> )
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> *ThreadPoolExecutor.java:1149*)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> *ThreadPoolExecutor.java:624*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*:
>>> Permission denied: no further information:
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> Caused by: *java.net.SocketException*: Permission denied: no further
>>> information
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*
>>> )
>>>
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(
>>> *SocketChannelImpl.java:715*)
>>>
>>> at
>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(
>>> *NioSocketChannel.java:330*)
>>>
>>> at
>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(
>>> *AbstractNioChannel.java:334*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
>>> *NioEventLoop.java:702*)
>>>
>>> at
>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
>>> *NioEventLoop.java:650*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
>>> *NioEventLoop.java:576*)
>>>
>>> at io.netty.channel.nio.NioEventLoop.run(
>>> *NioEventLoop.java:493*)
>>>
>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(
>>> *SingleThreadEventExecutor.java:989*)
>>>
>>> at io.netty.util.internal.ThreadExecutorMap$2.run(
>>> *ThreadExecutorMap.java:74*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> 21/01/20 17:07:12 ERROR RetryingBlockFetcher: Exception while beginning
>>> fetch of 1 outstanding blocks (after 3 retries)
>>>
>>> *java.io.IOException*: Failed to connect to
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:253*)
>>>
>>> at
>>> org.apache.spark.network.client.TransportClientFactory.createClient(
>>> *TransportClientFactory.java:195*)
>>>
>>> at
>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(
>>> *NettyBlockTransferService.scala:122*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(
>>> *RetryingBlockFetcher.java:141*)
>>>
>>> at
>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0(
>>> *RetryingBlockFetcher.java:169*)
>>>
>>> at java.util.concurrent.Executors$RunnableAdapter.call(
>>> *Executors.java:511*)
>>>
>>> at java.util.concurrent.FutureTask.run(*FutureTask.java:266*
>>> )
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> *ThreadPoolExecutor.java:1149*)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> *ThreadPoolExecutor.java:624*)
>>>
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(
>>> *FastThreadLocalRunnable.java:30*)
>>>
>>> at java.lang.Thread.run(*Thread.java:748*)
>>>
>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*:
>>> Permission denied: no further information:
>>> del1-lhp-n99999.synapse.com/192.168.166.213:51348
>>>
>>> Caused by: *java.net.SocketException*: Permission denied: no further
>>> information
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*
>>> )
>>>
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(
>>> *SocketChannelImpl.java:715*)
>>>
>>> at
>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(
>>> *NioSocketC*
>>>
>>
