I am getting this error some times when I run pyspark with spark 2.0.0 App > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) App > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) App > at java.lang.reflect.Method.invoke(Method.java:497) App > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237) App > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) App > at py4j.Gateway.invoke(Gateway.java:280) App > at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128) App > at py4j.commands.CallCommand.execute(CallCommand.java:79) App > at py4j.GatewayConnection.run(GatewayConnection.java:211) App > at java.lang.Thread.run(Thread.java:745) App > Caused by: java.io.IOException: java.lang.IllegalArgumentException: requirement failed: Block broadcast_21_piece0 is already present in the MemoryStore App > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1284) App > at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) App > at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) App > at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) App > at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) App > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) App > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:67) App > at org.apache.spark.scheduler.Task.run(Task.scala:85) App > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) App > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) App > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) App > ... 1 more App > *Caused by: java.lang.IllegalArgumentException: requirement failed: Block broadcast_21_piece0 is already present in the MemoryStore* App > at scala.Predef$.require(Predef.scala:224) App > at org.apache.spark.storage.memory.MemoryStore.putBytes(MemoryStore.scala:144) App > at org.apache.spark.storage.BlockManager$$anonfun$doPutBytes$1.apply(BlockManager.scala:792) App > at org.apache.spark.storage.BlockManager$$anonfun$doPutBytes$1.apply(BlockManager.scala:761) App > at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866) App > at org.apache.spark.storage.BlockManager.doPutBytes(BlockManager.scala:761) App > at org.apache.spark.storage.BlockManager.putBytes(BlockManager.scala:740) App > at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:140) App > at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:125) App > at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:125) App > at scala.collection.immutable.List.foreach(List.scala:381) App > at org.apache.spark.broadcast.TorrentBroadcast.org $apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:125) App > at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:186) App > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1277) App > ... 11 more App > App > 16/10/10 20:01:59 main INFO SparkContext: sc.stop called from [SparkSubmit.exceptionExitHook[failure]] App > 16/10/10 20:01:59 main INFO Autoscaler: Shutting down Autoscaler!! App > 16/10/10 20:01:59 main INFO ServerConnector: Stopped ServerConnector@1797a20 {HTTP/1.1} {0.0.0.0:4040}
Thanks Sandesh