> Spark now shades its own protobuf dependency so protobuf 2.4.1 should't be > getting pulled in unless you are directly using akka yourself. Are you?
No i'm not. Although I see that protobuf libraries are directly pulled into the 0.9.0 assembly jar - I do see the shaded version as well. e.g. below for Message.class -bash-4.1$ jar -ftv ./assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-hadoop2.0.0-cdh4.2.1.jar | grep protobuf | grep /Message.class 478 Thu Jun 30 15:26:12 PDT 2011 com/google/protobuf/Message.class 508 Sat Dec 14 14:20:38 PST 2013 com/google/protobuf_spark/Message.class > Does your project have other dependencies that might be indirectly pulling in > protobuf 2.4.1? It would be helpful if you could list all of your > dependencies including the exact Spark version and other libraries. I did have another one which I moved to the end of classpath - even ran partial code without that dependency but it still failed whenever I use the jar with ScalaBuf dependency. Spark version is 0.9.0 ~Vipul On Mar 31, 2014, at 4:51 PM, Patrick Wendell <pwend...@gmail.com> wrote: > Spark now shades its own protobuf dependency so protobuf 2.4.1 should't be > getting pulled in unless you are directly using akka yourself. Are you? > > Does your project have other dependencies that might be indirectly pulling in > protobuf 2.4.1? It would be helpful if you could list all of your > dependencies including the exact Spark version and other libraries. > > - Patrick > > > On Sun, Mar 30, 2014 at 10:03 PM, Vipul Pandey <vipan...@gmail.com> wrote: > I'm using ScalaBuff (which depends on protobuf2.5) and facing the same issue. > any word on this one? > On Mar 27, 2014, at 6:41 PM, Kanwaldeep <kanwal...@gmail.com> wrote: > > > We are using Protocol Buffer 2.5 to send messages to Spark Streaming 0.9 > > with > > Kafka stream setup. I have protocol Buffer 2.5 part of the uber jar deployed > > on each of the spark worker nodes. > > The message is compiled using 2.5 but then on runtime it is being > > de-serialized by 2.4.1 as I'm getting the following exception > > > > java.lang.VerifyError (java.lang.VerifyError: class > > com.snc.sinet.messages.XServerMessage$XServer overrides final method > > getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;) > > java.lang.ClassLoader.defineClass1(Native Method) > > java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > > java.lang.ClassLoader.defineClass(ClassLoader.java:615) > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > > > > Suggestions on how I could still use ProtoBuf 2.5. Based on the article - > > https://spark-project.atlassian.net/browse/SPARK-995 we should be able to > > use different version of protobuf in the application. > > > > > > > > > > > > -- > > View this message in context: > > http://apache-spark-user-list.1001560.n3.nabble.com/Using-ProtoBuf-2-5-for-messages-with-Spark-Streaming-tp3396.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > >
