Hi, I'm having trouble reading Bzip2 compressed sequence files after I enabled hadoop native libraries in spark.
Running LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/ $SPARK_HOME/bin/spark-submit --class .... gives the following error 5/06/26 00:48:02 INFO CodecPool: Got brand-new decompressor [.bz2] 15/06/26 00:48:02 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID 3) java.lang.UnsupportedOperationException at org.apache.hadoop.io.compress.bzip2.BZip2DummyDecompressor.decompress(BZip2DummyDecompressor.java:32) at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91) at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) at java.io.DataInputStream.readFully(DataInputStream.java:195) at java.io.DataInputStream.readLong(DataInputStream.java:416) removing the LD_LIBRARY_PATH makes spark run fine but it gives the following warning WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Has anyone else run into this issue ? Any help is welcome.