But I am working on my local development machine, so it should have no
relative to workers/executers.
I find some documents about enable snappy on hadoop. If I want to use
snappy with spark, do I need to config spark as hadoop or have some easy
way to access it?
Regard,
Junfeng Chen
On Mon, Dec 4, 2017 at 4:12 PM, Qiao, Richard <richard.q...@capitalone.com>
wrote:
> It seems a common mistake that the path is not accessible by
> workers/executors.
>
> Best regards
> Richard
>
> Sent from my iPhone
>
> On Dec 3, 2017, at 22:32, Junfeng Chen <darou...@gmail.com> wrote:
>
> I am working on importing snappy compressed json file into spark rdd or
> dataset. However I meet this error: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>
> I have set the following configuration:
>
> SparkConf conf = new SparkConf()
> .setAppName("normal spark")
> .setMaster("local")
> .set("spark.io.compression.codec",
> "org.apache.spark.io.SnappyCompressionCodec")
>
> .set("spark.driver.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
>
> .set("spark.driver.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
>
> .set("spark.executor.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
>
> .set("spark.executor.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
> ;
>
> Where D:\Downloads\spark-2.2.0-bin-hadoop2.7 is my spark unpacked path,
> and I can find the snappy jar file snappy-0.2.jar and
> snappy-java-1.1.2.6.jar in
>
> D:\Downloads\spark-2.2.0-bin-hadoop2.7\spark-2.2.0-bin-hadoop2.7\jars\
>
> However nothing works and even the error message not change.
>
> How can I fix it?
>
>
> ref of stackoverflow: https://stackoverflow.com/questions/47626012/
> config-snappy-support-for-spark-in-windows
> <https://stackoverflow.com/questions/47626012/config-snappy-support-for-spark-in-windows>
>
>
>
> Regard,
> Junfeng Chen
>
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
