Do you have a list/array in your avro record? If yes this could cause the problem. I experienced this kind of problem and solved it by providing custom kryo ser/de for avro lists. Also be carefull spark reuses records, so if you just read and then don't copy/transform them you would end up with the records having same values.
2014-07-22 15:01 GMT+02:00 Sparky <gullo_tho...@bah.com>: > Running a simple collect method on a group of Avro objects causes a plain > NullPointerException. Does anyone know what may be wrong? > > >files.collect() > > Press ENTER or type command to continue > Exception in thread "Executor task launch worker-0" > java.lang.NullPointerException > at > > org.apache.spark.executor.Executor$TaskRunner$$anonfun$2.apply(Executor.scala:254) > at > > org.apache.spark.executor.Executor$TaskRunner$$anonfun$2.apply(Executor.scala:254) > at scala.Option.flatMap(Option.scala:170) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:254) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/collect-on-small-group-of-Avro-files-causes-plain-NullPointerException-tp10400.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
