Protobuf java code generated by ptotoc 2.4 does not compile with protobuf library 2.5 - that's true. What I meant: You serialized message with class generated with protobuf 2.4.1. Now you can read that message with class generated with protobuf 2.5.0 from same .proto.
2014-03-01 0:00 GMT+04:00 Egor Pahomov <pahomov.e...@gmail.com>: > In that same pom > > <profile> > <id>yarn</id> > > <properties> > <hadoop.major.version>2</hadoop.major.version> > <hadoop.version>2.2.0</hadoop.version> > <protobuf.version>2.5.0</protobuf.version> > </properties> > > <modules> > <module>yarn</module> > </modules> > > </profile> > > > > 2014-02-28 23:46 GMT+04:00 Aureliano Buendia <buendia...@gmail.com>: > > >> >> >> On Fri, Feb 28, 2014 at 7:17 PM, Egor Pahomov <pahomov.e...@gmail.com>wrote: >> >>> Spark 0.9 uses protobuf 2.5.0 >>> >> >> Spark 0.9 uses 2.4.1: >> >> >> https://github.com/apache/incubator-spark/blob/4d880304867b55a4f2138617b30600b7fa013b14/pom.xml#L118 >> >> Is there another pom for when hadoop 2.2 is used? I don't see another >> branch for hadooop 2.2. >> >> >>> Hadoop 2.2 uses protobuf 2.5.0 >>> protobuf 2.5.0 can read massages serialized with protobuf 2.4.1 >>> >> >> Protobuf java code generated by ptotoc 2.4 does not compile with protobuf >> library 2.5. This is what the OP's error message is about. >> >> >>> So there is not any reason why you can't read some messages from hadoop >>> 2.2 with protobuf 2.5.0, probably you somehow have 2.4.1 in your class >>> path. Of course it's very bad, that you have both 2.4.1 and 2.5.0 in your >>> classpath. Use excludes or whatever to get rid of 2.4.1. >>> >>> Personally, I spend 3 days to move my project to protobuf 2.5.0 from >>> 2.4.1. But it has to be done for the whole your project. >>> >>> 2014-02-28 21:49 GMT+04:00 Aureliano Buendia <buendia...@gmail.com>: >>> >>> Doesn't hadoop 2.2 also depend on protobuf 2.4? >>>> >>>> >>>> On Fri, Feb 28, 2014 at 5:45 PM, Ognen Duzlevski < >>>> og...@plainvanillagames.com> wrote: >>>> >>>>> A stupid question, by the way, you did compile Spark with Hadoop 2.2.0 >>>>> support? >>>>> >>>>> Ognen >>>>> >>>>> On 2/28/14, 10:51 AM, Prasad wrote: >>>>> >>>>>> Hi >>>>>> I am getting the protobuf error.... while reading HDFS file using >>>>>> spark >>>>>> 0.9.0 -- i am running on hadoop 2.2.0 . >>>>>> >>>>>> When i look thru, i find that i have both 2.4.1 and 2.5 and some blogs >>>>>> suggest that there is some incompatability issues betwen 2.4.1 and 2.5 >>>>>> >>>>>> hduser@prasadHdp1:~/spark-0.9.0-incubating$ find ~/ -name >>>>>> protobuf-java*.jar >>>>>> /home/hduser/.m2/repository/com/google/protobuf/protobuf- >>>>>> java/2.4.1/protobuf-java-2.4.1.jar >>>>>> /home/hduser/.m2/repository/org/spark-project/protobuf/ >>>>>> protobuf-java/2.4.1-shaded/protobuf-java-2.4.1-shaded.jar >>>>>> /home/hduser/spark-0.9.0-incubating/lib_managed/ >>>>>> bundles/protobuf-java-2.5.0.jar >>>>>> /home/hduser/spark-0.9.0-incubating/lib_managed/jars/ >>>>>> protobuf-java-2.4.1-shaded.jar >>>>>> /home/hduser/.ivy2/cache/com.google.protobuf/protobuf-java/ >>>>>> bundles/protobuf-java-2.5.0.jar >>>>>> /home/hduser/.ivy2/cache/org.spark-project.protobuf/ >>>>>> protobuf-java/jars/protobuf-java-2.4.1-shaded.jar >>>>>> >>>>>> >>>>>> Can someone please let me know if you faced these issues and how u >>>>>> fixed it. >>>>>> >>>>>> Thanks >>>>>> Prasad. >>>>>> Caused by: java.lang.VerifyError: class >>>>>> org.apache.hadoop.security.proto.SecurityProtos$ >>>>>> GetDelegationTokenRequestProto >>>>>> overrides final method >>>>>> getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; >>>>>> at java.lang.ClassLoader.defineClass1(Native Method) >>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800) >>>>>> at >>>>>> java.security.SecureClassLoader.defineClass( >>>>>> SecureClassLoader.java:142) >>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader. >>>>>> java:449) >>>>>> at java.net.URLClassLoader.access$100(URLClassLoader. >>>>>> java:71) >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>>>> at java.security.AccessController.doPrivileged(Native >>>>>> Method) >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java: >>>>>> 354) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>>>> at java.lang.Class.getDeclaredMethods0(Native Method) >>>>>> at java.lang.Class.privateGetDeclaredMethods( >>>>>> Class.java:2531) >>>>>> at java.lang.Class.privateGetPublicMethods(Class.java:2651) >>>>>> at java.lang.Class.privateGetPublicMethods(Class.java:2661) >>>>>> at java.lang.Class.getMethods(Class.java:1467) >>>>>> at >>>>>> sun.misc.ProxyGenerator.generateClassFile(ProxyGenerator.java:426) >>>>>> at >>>>>> sun.misc.ProxyGenerator.generateProxyClass(ProxyGenerator.java:323) >>>>>> at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:636) >>>>>> at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:722) >>>>>> at >>>>>> org.apache.hadoop.ipc.ProtobufRpcEngine.getProxy( >>>>>> ProtobufRpcEngine.java:92) >>>>>> at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:537) >>>>>> >>>>>> >>>>>> Caused by: java.lang.reflect.InvocationTargetException >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>> Method) >>>>>> at >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke( >>>>>> NativeMethodAccessorImpl.java:57) >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke( >>>>>> DelegatingMethodAccessorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: http://apache-spark-user-list. >>>>>> 1001560.n3.nabble.com/Error-reading-HDFS-file-using-spark- >>>>>> 0-9-0-hadoop-2-2-0-incompatible-protobuf-2-5-and-2-4-1-tp2158.html >>>>>> Sent from the Apache Spark User List mailing list archive at >>>>>> Nabble.com. >>>>>> >>>>> >>>>> -- >>>>> Some people, when confronted with a problem, think "I know, I'll use >>>>> regular expressions." Now they have two problems. >>>>> -- Jamie Zawinski >>>>> >>>>> >>>> >>> >>> >>> -- >>> >>> >>> >>> *Sincerely yours Egor PakhomovScala Developer, Yandex* >>> >> >> > > > -- > > > > *Sincerely yours Egor PakhomovScala Developer, Yandex* > -- *Sincerely yoursEgor PakhomovScala Developer, Yandex*
