So it looks like shark introduces protobuf-2.4, are you sure shark depends on the right spark version ? I am not familiar with shark's dependencies, but in your case, you could delete the protobuf-2.4 and it should not affect anything.
On Wed, Mar 26, 2014 at 6:24 PM, qingyang li <liqingyang1...@gmail.com>wrote: > my spark can also work well with hadoop2.2.0 > my shark can not work well with hadoop2.2.0 because protobuf version > problem. > > in shark direcotry , i found two vesions of protobuf, and they all are > loaded into classpath. > > [root@bigdata001 shark-0.9.0]# find . -name "proto*.jar" > > > ./lib_managed/jars/org.spark-project.protobuf/protobuf-java/protobuf-java-2.4.1-shaded.jar > > ./lib_managed/bundles/com.google.protobuf/protobuf-java/protobuf-java-2.5.0.jar > > > > > 2014-03-27 2:26 GMT+08:00 yao <yaosheng...@gmail.com>: > > > @qingyang, spark 0.9.0 works for me perfectly when accessing (read/write) > > data on hdfs. BTW, if you look at pom.xml, you have to choose yarn > profile > > to compile spark, so that it won't include protobuf 2.4.1 in your final > > jars. Here is the command line we use to compile spark with hadoop 2.2: > > > > mvn -U -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0 -Pyarn -DskipTests > > package > > > > Thanks > > -Shengzhe > > > > > > On Wed, Mar 26, 2014 at 12:04 AM, qingyang li <liqingyang1...@gmail.com > > >wrote: > > > > > Egor, i encounter the same problem which you have asked in this thread: > > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/spark-user/201402.mbox/%3CCAMrx5DwJVJS0g_FE7_2qwMu4Xf0y5VfV=tlyauv2kh5v4k6...@mail.gmail.com%3E > > > > > > have you fixed this problem? > > > > > > i am using shark to read a table which i have created on hdfs. > > > > > > i found in shark lib_managed directory there are two protobuf*.jar: > > > [root@bigdata001 shark-0.9.0]# find . -name "proto*.jar" > > > > > > > > > ./lib_managed/jars/org.spark-project.protobuf/protobuf-java/protobuf-java-2.4.1-shaded.jar > > > > > > > > > ./lib_managed/bundles/com.google.protobuf/protobuf-java/protobuf-java-2.5.0.jar > > > > > > > > > my hadoop is using protobuf-java-2.5.0.jar . > > > > > >
