I have encountered the same issues. before I changed the spark version i setted
up environment as follows.
spark 1.5.2
hadoop 2.6.2
hive 1.2.1
but no luck it's not work well, even through i run essembly hive in spark with
jdbc mode there is also some properblems.
then I changed the spark version 1.3.1 and rebuild, I just run a example
,there is a issuse seems the pb version conflict, will rebuild and try
again.....
Exception in thread "main" java.lang.VerifyError: class
org.apache.hadoop.yarn.proto.YarnProtos$PriorityProto overrides final method
getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; at
java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:800) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
------------------ Original ------------------
From: "Mich Talebzadeh";<m...@peridale.co.uk>;
Date: Thu, Dec 3, 2015 06:28 PM
To: "user"<u...@hive.apache.org>; "user"<user@spark.apache.org>;
Subject: Building spark 1.3 from source code to work with Hive 1.2.1
Hi,
I have seen mails that state that the user has managed to build spark 1.3 to
work with Hive. I tried Spark 1.5.2 but no luck
I downloaded spark source 1.3 source code spark-1.3.0.tar and built it as
follows
./make-distribution.sh --name "hadoop2-without-hive" --tgz
"-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"
This successfully completed and created the tarred zip file. I then created
spark 1.3 tree from this zipped file. $SPARK_HOME is /usr/lib/spark
Other steps that I performed:
1. In $HIVE_HOME/lib , I copied spark-assembly-1.3.0-hadoop2.4.0.jar to
this directory
2. In $SPARK_HOME/conf I created a syblink to /usr/lib/hive/conf/hive-site.xml
Then I tried to start spark master node
/usr/lib/spark/sbin/start-master.sh
I get the following error:
cat
/usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out
Spark Command: /usr/java/latest/bin/java -cp
:/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/home/hduser/hadoop-2.6.0/etc/hadoop
-XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m
org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port 8080
========================================
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
at java.lang.Class.getMethod0(Class.java:2764)
at java.lang.Class.getMethod(Class.java:1653)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I also notice that in /usr/lib/spark/lib, I only have the following jar files
-rw-r--r-- 1 hduser hadoop 98795479 Dec 3 09:03
spark-examples-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 98187168 Dec 3 09:03
spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-r--r-- 1 hduser hadoop 4136760 Dec 3 09:03 spark-1.3.0-yarn-shuffle.jar
Wheras in pre-build downloaded one à /usr/lib/spark-1.3.0-bin-hadoop2.4, there
are additional JAR files
-rw-rw-r-- 1 hduser hadoop 1890075 Mar 6 2015 datanucleus-core-3.2.10.jar
-rw-rw-r-- 1 hduser hadoop 112446389 Mar 6 2015
spark-examples-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 159319006 Mar 6 2015
spark-assembly-1.3.0-hadoop2.4.0.jar
-rw-rw-r-- 1 hduser hadoop 4136744 Mar 6 2015 spark-1.3.0-yarn-shuffle.jar
-rw-rw-r-- 1 hduser hadoop 1809447 Mar 6 2015 datanucleus-rdbms-3.2.9.jar
-rw-rw-r-- 1 hduser hadoop 339666 Mar 6 2015 datanucleus-api-jdo-3.2.6.jar
Any ideas what is is missing? I am sure someone has sorted this one out before.
Thanks,
Mich
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this message
shall not be understood as given or endorsed by Peridale Technology Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept any
responsibility.
