Thanks for the super quick response!

I can read the file just fine in hadoop, it's just when I point Spark at
this file it can't seem to read it due to the missing snappy jars / so's.

I'l paying around with adding some things to spark-env.sh file, but still
nothing.

On Wed, Apr 1, 2015 at 7:19 PM, Xianjin YE <advance...@gmail.com> wrote:

> Can you read snappy compressed file in hdfs?  Looks like the libsnappy.so
> is not in the hadoop native lib path.
>
> On Thursday, April 2, 2015 at 10:13 AM, Nick Travers wrote:
>
> Has anyone else encountered the following error when trying to read a
> snappy
> compressed sequence file from HDFS?
>
> *java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z*
>
> The following works for me when the file is uncompressed:
>
> import org.apache.hadoop.io._
> val hdfsPath = "hdfs://nost.name/path/to/folder"
> val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
> file.count()
>
> but fails when the encoding is Snappy.
>
> I've seen some stuff floating around on the web about having to explicitly
> enable support for Snappy in spark, but it doesn't seem to work for me:
> http://www.ericlin.me/enabling-snappy-support-for-sharkspark
> <http://www.ericlin.me/enabling-snappy-support-for-sharkspark>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-snappy-and-HDFS-tp22349.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
>

Reply via email to