[ https://issues.apache.org/jira/browse/PIG-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liyunzhang_intel updated PIG-4859: ---------------------------------- Attachment: PIG-4859.patch [~pallavi.rao], [~mohitsabharwal],[~kexianda]: Changes in PIG-4859.patch 1. upgrade snappy-java to 1.1.1.3 in ivy/libraries > Need upgrade snappy-java.version to 1.1.1.3 > ------------------------------------------- > > Key: PIG-4859 > URL: https://issues.apache.org/jira/browse/PIG-4859 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: liyunzhang_intel > Assignee: liyunzhang_intel > Fix For: spark-branch > > Attachments: PIG-4859.patch > > > run pig on spark on yarn-client env as following: > export SPARK_MASTER="yarn-client" > ./pig -x spark xxx.pig > Throw error like following: > {code} > main] 2016-03-30 16:52:26,115 INFO scheduler.DAGScheduler > (Logging.scala:logInfo(59)) - Job 0 failed: saveAsNewAPIHadoopDataset at > StoreConverter.java:101, took 73.980147 s > 19895 [main] 2016-03-30 16:52:26,119 ERROR spark.JobGraphBuilder > (JobGraphBuilder.java:sparkOperToRDD(166)) - throw exception in > sparkOperToRDD: > 19896 org.apache.spark.SparkException: Job aborted due to stage failure: Task > 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage > 0.0 (TID 3, zly1.sh.intel.com): java .lang.UnsatisfiedLinkError: > org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I > 19897 at org.xerial.snappy.SnappyNative.uncompressedLength(Native > Method) > 19898 at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:541) > 19899 at > org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:350) > 19900 at > org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158) > 19901 at > org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142) > 19902 at > java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2313) > 19903 at > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2326) > 19904 at > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797) > 19905 at > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) > 19906 at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > 19907 at > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:64) > 19908 at > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:64) > 19909 at > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:103) > 19910 at > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:216) > 19911 at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)