Can you try increasing the partition for the base RDD/dataframe that you are working on?
On Tue, May 8, 2018 at 5:05 PM, Debabrata Ghosh <mailford...@gmail.com> wrote: > Hi Everyone, > I have been trying to run spark-shell in YARN client mode, but am getting > lot of ClosedChannelException errors, however the program works fine on > local mode. I am using spark 2.2.0 build for Hadoop 2.7.3. If you are > familiar with this error, please can you help with the possible resolution. > > Any help would be greatly appreciated! > > Here is the error message: > > 18/05/08 00:01:18 ERROR TransportClient: Failed to send RPC > 7905321254854295784 to /9.30.94.43:60220: java.nio.channels. > ClosedChannelException > java.nio.channels.ClosedChannelException > at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown > Source) > 18/05/08 00:01:18 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: > Sending RequestExecutors(5,0,Map(),Set()) to AM was unsuccessful > java.io.IOException: Failed to send RPC 7905321254854295784 to / > 9.30.94.43:60220: java.nio.channels.ClosedChannelException > at org.apache.spark.network.client.TransportClient.lambda$ > sendRpc$2(TransportClient.java:237) > at io.netty.util.concurrent.DefaultPromise.notifyListener0( > DefaultPromise.java:507) > at io.netty.util.concurrent.DefaultPromise.notifyListenersNow( > DefaultPromise.java:481) > at io.netty.util.concurrent.DefaultPromise.access$000( > DefaultPromise.java:34) > at io.netty.util.concurrent.DefaultPromise$1.run( > DefaultPromise.java:431) > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks( > SingleThreadEventExecutor.java:399) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) > at io.netty.util.concurrent.SingleThreadEventExecutor$2. > run(SingleThreadEventExecutor.java:131) > at io.netty.util.concurrent.DefaultThreadFactory$ > DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.channels.ClosedChannelException > > Cheers, > > Debu > -- Thanks Deepak www.bigdatabig.com www.keosha.net
