liyunzhang_intel created PIG-4859:
-------------------------------------
Summary: Need upgrade snappy-java.version to 1.1.1.3
Key: PIG-4859
URL: https://issues.apache.org/jira/browse/PIG-4859
Project: Pig
Issue Type: Sub-task
Components: spark
Reporter: liyunzhang_intel
Assignee: liyunzhang_intel
run pig on spark on yarn-client env as following:
export SPARK_MASTER="yarn-client"
./pig -x spark xxx.pig
Throw error like following:
{code}
main] 2016-03-30 16:52:26,115 INFO scheduler.DAGScheduler
(Logging.scala:logInfo(59)) - Job 0 failed: saveAsNewAPIHadoopDataset at
StoreConverter.java:101, took 73.980147 s
19895 [main] 2016-03-30 16:52:26,119 ERROR spark.JobGraphBuilder
(JobGraphBuilder.java:sparkOperToRDD(166)) - throw exception in sparkOperToRDD:
19896 org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0
(TID 3, zly1.sh.intel.com): java .lang.UnsatisfiedLinkError:
org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I
19897 at org.xerial.snappy.SnappyNative.uncompressedLength(Native
Method)
19898 at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:541)
19899 at
org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:350)
19900 at
org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158)
19901 at
org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
19902 at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2313)
19903 at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2326)
19904 at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797)
19905 at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
19906 at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
19907 at
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:64)
19908 at
org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:64)
19909 at
org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:103)
19910 at
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:216)
19911 at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
