Do you think this binary would have issue? Do I need to build spark from source code?
On Tue, Aug 25, 2015 at 1:06 PM, Cassa L <lcas...@gmail.com> wrote: > I downloaded below binary version of spark. > spark-1.4.1-bin-cdh4 > > On Tue, Aug 25, 2015 at 1:03 PM, java8964 <java8...@hotmail.com> wrote: > >> Did your spark build with Hive? >> >> I met the same problem before because the hive-exec jar in the maven >> itself include "protobuf" class, which will be included in the Spark jar. >> >> Yong >> >> ------------------------------ >> Date: Tue, 25 Aug 2015 12:39:46 -0700 >> Subject: Re: Protobuf error when streaming from Kafka >> From: lcas...@gmail.com >> To: yuzhih...@gmail.com >> CC: user@spark.apache.org >> >> >> Hi, >> I am using Spark-1.4 and Kafka-0.8.2.1 >> As per google suggestions, I rebuilt all the classes with protobuff-2.5 >> dependencies. My new protobuf is compiled using 2.5. However now, my spark >> job does not start. Its throwing different error. Does Spark or any other >> its dependencies uses old protobuff-2.4? >> >> Exception in thread "main" java.lang.VerifyError: class >> com.apple.ist.retail.xcardmq.serializers.SampleProtobufMessage$ProtoBuff >> overrides final method >> getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:760) >> at >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) >> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:362) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:361) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> at >> com.apple.ist.retail.spark.kafka.consumer.SparkMQProcessor.start(SparkProcessor.java:68) >> at >> com.apple.ist.retail.spark.kafka.consumer.SparkMQConsumer.main(SparkConsumer.java:43) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665) >> at >> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> >> On Mon, Aug 24, 2015 at 6:53 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >> Can you show the complete stack trace ? >> >> Which Spark / Kafka release are you using ? >> >> Thanks >> >> On Mon, Aug 24, 2015 at 4:58 PM, Cassa L <lcas...@gmail.com> wrote: >> >> Hi, >> I am storing messages in Kafka using protobuf and reading them into >> Spark. I upgraded protobuf version from 2.4.1 to 2.5.0. I got >> "java.lang.UnsupportedOperationException" for older messages. However, even >> for new messages I get the same error. Spark does convert it though. I see >> my messages. How do I get rid of this error? >> java.lang.UnsupportedOperationException: This is supposed to be >> overridden by subclasses. >> at >> com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180) >> at >> org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407) >> at >> com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749) >> >> >> >> >
