Dear Spark Users:
Spark 1.0 has been installed as Standalone - But it can't read any compressed
(CMX/Snappy) and Sequence file residing on HDFS (it can read uncompressed files
from HDFS). The key notable message is: Unable to load native-hadoop
library.. Other related messages are -
I’d suggest asking the IBM Hadoop folks, but my guess is that the library
cannot be found in /opt/IHC/lib/native/Linux-amd64-64/. Or maybe if this
exception is happening in your driver program, the driver program’s
java.library.path doesn’t include this. (SPARK_LIBRARY_PATH from spark-env.sh
