Hi,
I have been following Aniket's spork github repository.
https://github.com/aniket486/pig
I have done all the changes mentioned in recently modified pig-spark file.
I am using:
hadoop 2.0.5 alpha
spark-0.8.1-incubating
mesos 0.16.0
##PIG variables
export *HADOOP_CONF_DIR*=$HADOOP_INSTALL/etc/hadoop
export *SPARK_YARN_APP_JAR*=/home/ubuntu/pig/pig-withouthadoop.jar
export *SPARK_JAVA_OPTS*=" -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/heap.dump"
export
*SPARK_JAR*=/home/ubuntu/spark/assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.0.5-alpha.jar
export *SPARK_MASTER*=yarn-client
export *SPARK_HOME*=/home/ubuntu/spark
export *SPARK_JARS*=/home/ubuntu/pig/contrib/piggybank/java/piggybank.jar
export
*PIG_CLASSPATH*=${SPARK_JAR}:${SPARK_JARS}:/home/ubuntu/mesos/build/src/mesos-0.16.0.jar:/home/ubuntu/pig/pig-withouthadoop.jar
export *SPARK_PIG_JAR*=/home/ubuntu/pig/pig-withouthadoop.jar
This works fine in Mapreduce and local mode. But, while running on spark
mode I am facing follwing error. This error come after the job is submitted
and run on yarn-master.
Can you please tell me how to proceed.
###########################################error
message############################################################
*ERROR 2998*: *Unhandled internal error. class
org.apache.spark.util.InnerClosureFinder* has interface
org.objectweb.asm.ClassVisitor as super class
*java.lang.IncompatibleClassChangeError*: class
org.apache.spark.util.InnerClosureFinder has interface
org.objectweb.asm.ClassVisitor as super class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
at
org.apache.spark.util.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:87)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:107)
at org.apache.spark.SparkContext.clean(SparkContext.scala:970)
at org.apache.spark.rdd.RDD.map(RDD.scala:246)
at
org.apache.pig.backend.hadoop.executionengine.spark.converter.LoadConverter.convert(LoadConverter.java:68)
at
org.apache.pig.backend.hadoop.executionengine.spark.converter.LoadConverter.convert(LoadConverter.java:38)
at
org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:212)
at
org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:201)
at
org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:201)
at
org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:125)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1328)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1310)
at org.apache.pig.PigServer.storeEx(PigServer.java:993)
at org.apache.pig.PigServer.store(PigServer.java:957)
at org.apache.pig.PigServer.openIterator(PigServer.java:870)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:729)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:370)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:609)
at org.apache.pig.Main.main(Main.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
================================================================================
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Pig-on-Spark-tp2367p3187.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
