You can run hadoop checknative -a and see if bzip2 is detected correctly.
-- Ruslan Dautkhanov On Fri, Jun 26, 2015 at 10:18 AM, Marcelo Vanzin <van...@cloudera.com> wrote: > What master are you using? If this is not a "local" master, you'll need to > set LD_LIBRARY_PATH on the executors also (using > spark.executor.extraLibraryPath). > > If you are using local, then I don't know what's going on. > > On Fri, Jun 26, 2015 at 1:39 AM, Arunabha Ghosh <arunabha...@gmail.com> > wrote: > >> Hi, >> I'm having trouble reading Bzip2 compressed sequence files after I >> enabled hadoop native libraries in spark. >> >> Running >> LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/ $SPARK_HOME/bin/spark-submit >> --class .... gives the following error >> >> 5/06/26 00:48:02 INFO CodecPool: Got brand-new decompressor [.bz2] >> 15/06/26 00:48:02 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID >> 3) >> java.lang.UnsupportedOperationException >> at >> org.apache.hadoop.io.compress.bzip2.BZip2DummyDecompressor.decompress(BZip2DummyDecompressor.java:32) >> at >> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91) >> at >> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) >> at java.io.DataInputStream.readFully(DataInputStream.java:195) >> at java.io.DataInputStream.readLong(DataInputStream.java:416) >> >> removing the LD_LIBRARY_PATH makes spark run fine but it gives the >> following warning >> WARN NativeCodeLoader: Unable to load native-hadoop library for your >> platform... using builtin-java classes where applicable >> >> Has anyone else run into this issue ? Any help is welcome. >> > > > > -- > Marcelo >
