What master are you using? If this is not a "local" master, you'll need to set LD_LIBRARY_PATH on the executors also (using spark.executor.extraLibraryPath).
If you are using local, then I don't know what's going on. On Fri, Jun 26, 2015 at 1:39 AM, Arunabha Ghosh <arunabha...@gmail.com> wrote: > Hi, > I'm having trouble reading Bzip2 compressed sequence files after I > enabled hadoop native libraries in spark. > > Running > LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/ $SPARK_HOME/bin/spark-submit > --class .... gives the following error > > 5/06/26 00:48:02 INFO CodecPool: Got brand-new decompressor [.bz2] > 15/06/26 00:48:02 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID > 3) > java.lang.UnsupportedOperationException > at > org.apache.hadoop.io.compress.bzip2.BZip2DummyDecompressor.decompress(BZip2DummyDecompressor.java:32) > at > org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readLong(DataInputStream.java:416) > > removing the LD_LIBRARY_PATH makes spark run fine but it gives the > following warning > WARN NativeCodeLoader: Unable to load native-hadoop library for your > platform... using builtin-java classes where applicable > > Has anyone else run into this issue ? Any help is welcome. > -- Marcelo
