That converts the error to the following
14/10/08 13:27:40 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 14/10/08 13:27:40 INFO broadcast.HttpBroadcast: Started reading broadcast variable 0 14/10/08 13:27:40 ERROR executor.Executor: Exception in task 1.0 in stage 0.0 (TID 1) java.io.FileNotFoundException: http://192.168.1.4:54221/broadcast_0 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1610) at org.apache.spark.broadcast.HttpBroadcast$.org$apache$spark$broadcast$HttpBroadcast$$read(HttpBroadcast.scala:197) at org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:89) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:991) Curiously the error is very repeatable on a relatively large and complex program I am running but the same Spark steps work well when the Objects are Strings and Integers like word count. My objects are complex but Serialize well and run when I drop a combineByKey step On Wed, Oct 8, 2014 at 12:00 PM, Liquan Pei <liquan...@gmail.com> wrote: > Hi Lewis, > > For debugging purpose, can you try using HttpBroadCast to see if the error > remains? You can enable HttpBroadCast by setting spark.broadcast.factory > to org.apache.spark.broadcast.HttpBroadcastFactory in spark conf. > > Thanks, > Liquan > > On Wed, Oct 8, 2014 at 11:21 AM, Steve Lewis <lordjoe2...@gmail.com> > wrote: > >> I am running on Windows 8 using Spark 1.1.0 in local mode with Hadoop 2.2 >> - I repeatedly see >> the following in my logs. >> >> I believe this happens in combineByKey >> >> >> 14/10/08 09:36:30 INFO executor.Executor: Running task 3.0 in stage 0.0 >> (TID 3) >> 14/10/08 09:36:30 INFO broadcast.TorrentBroadcast: Started reading >> broadcast variable 0 >> 14/10/08 09:36:35 ERROR broadcast.TorrentBroadcast: Reading broadcast >> variable 0 failed >> 14/10/08 09:36:35 INFO broadcast.TorrentBroadcast: Reading broadcast >> variable 0 took 5.006378813 s >> 14/10/08 09:36:35 INFO broadcast.TorrentBroadcast: Started reading >> broadcast variable 0 >> 14/10/08 09:36:35 ERROR executor.Executor: Exception in task 0.0 in stage >> 0.0 (TID 0) >> java.lang.NullPointerException >> at java.nio.ByteBuffer.wrap(ByteBuffer.java:392) >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58) >> at org.apache.spark.scheduler.Task.run(Task.scala:54) >> >> - >> > > > > -- > Liquan Pei > Department of Physics > University of Massachusetts Amherst > -- Steven M. Lewis PhD 4221 105th Ave NE Kirkland, WA 98033 206-384-1340 (cell) Skype lordjoe_com
