Re: keep or remove sc.stop() coz of RpcEnv already stopped error
Hi , I am using spark-1.6 how to ignore this warning because of this Illegal state exception my production jobs which are scheduld are showing completed abnormally... I cant even handle exception as after sc.stop if i try to execute any code again this exception will come from catch block.. so i removed sc.stop completely and let it exit on its own is it okay? On Mon, Mar 13, 2017 at 7:01 PM, Yong Zhang wrote: > What version of Spark you are using? > > > Based on Spark-12967, it is fixed on Spark 2.0 and later. If you are using > Spark 1.x, you can ignore this Warning. It shouldn't affect any functions. > > > Yong > > -- > *From:* nancy henry > *Sent:* Monday, March 13, 2017 7:08 AM > *To:* user @spark > *Subject:* keep or remove sc.stop() coz of RpcEnv already stopped error > > > Hi Team, > > > getting this error if we put sc.stop() in application.. > > > > can we remove it from application but i read if yu dont explicitly stop > using sc.stop the yarn application will not get registered in history > service.. SO what to do ? > > > WARN Dispatcher: Message RemoteProcessDisconnected dropped. > > java.lang.IllegalStateException: RpcEnv already stopped. > > at org.apache.spark.rpc.netty.Dispatcher.postMessage( > Dispatcher.scala:159) > > at org.apache.spark.rpc.netty.Dispatcher.postToAll( > Dispatcher.scala:109) > > at org.apache.spark.rpc.netty.NettyRpcHandler. > connectionTerminated(NettyRpcEnv.scala:630) > > at org.apache.spark.network.server.TransportRequestHandler. > channelUnregistered(TransportRequestHandler.java:94) > > > > >
Re: keep or remove sc.stop() coz of RpcEnv already stopped error
What version of Spark you are using? Based on Spark-12967, it is fixed on Spark 2.0 and later. If you are using Spark 1.x, you can ignore this Warning. It shouldn't affect any functions. Yong From: nancy henry Sent: Monday, March 13, 2017 7:08 AM To: user @spark Subject: keep or remove sc.stop() coz of RpcEnv already stopped error Hi Team, getting this error if we put sc.stop() in application.. can we remove it from application but i read if yu dont explicitly stop using sc.stop the yarn application will not get registered in history service.. SO what to do ? WARN Dispatcher: Message RemoteProcessDisconnected dropped. java.lang.IllegalStateException: RpcEnv already stopped. at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:159) at org.apache.spark.rpc.netty.Dispatcher.postToAll(Dispatcher.scala:109) at org.apache.spark.rpc.netty.NettyRpcHandler.connectionTerminated(NettyRpcEnv.scala:630) at org.apache.spark.network.server.TransportRequestHandler.channelUnregistered(TransportRequestHandler.java:94)
keep or remove sc.stop() coz of RpcEnv already stopped error
Hi Team, getting this error if we put sc.stop() in application.. can we remove it from application but i read if yu dont explicitly stop using sc.stop the yarn application will not get registered in history service.. SO what to do ? WARN Dispatcher: Message RemoteProcessDisconnected dropped. java.lang.IllegalStateException: RpcEnv already stopped. at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:159) at org.apache.spark.rpc.netty.Dispatcher.postToAll(Dispatcher.scala:109) at org.apache.spark.rpc.netty.NettyRpcHandler.connectionTerminated(NettyRpcEnv.scala:630) at org.apache.spark.network.server.TransportRequestHandler.channelUnregistered(TransportRequestHandler.java:94)