about java.io.EOFException / java.lang.ClassNotFoundException: kafka.common.OffsetOutOfRangeException

[Selina Tech]

Hi:
I am processing on a new Kafka topic with Spark and then I got error
below. I google this questions, looks like I lot of people having similar
problems before. But I have not got clue yet.

   Is any one know how to fix this issue?

Sincerely.
Selina


00:39:58,004 WARN  - 2017-03-18
00:39:57,921:7726(0x7f22e69b8700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 28ms
00:41:01,298 WARN  - 17/03/18 00:41:01 WARN Selector: Error in I/O with /
10.128.64.152
00:41:01,298 WARN  - java.io.EOFException
00:41:01,298 WARN  - at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62)
00:41:01,298 WARN  - at
org.apache.kafka.common.network.Selector.poll(Selector.java:248)
00:41:01,298 WARN  - at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192)
00:41:01,298 WARN  - at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
00:41:01,298 WARN  - at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
00:41:01,298 WARN  - at java.lang.Thread.run(Thread.java:745)
00:43:31,514 WARN  - 2017-03-18
00:43:31,514:7726(0x7f22e69b8700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 11ms
00:44:17,996 WARN  - 17/03/18 00:44:17 WARN ThrowableSerializationWrapper:
Task exception could not be deserialized
00:44:17,996 WARN  - java.lang.ClassNotFoundException:
kafka.common.OffsetOutOfRangeException
00:44:17,996 WARN  - at
java.net.URLClassLoader$1.run(URLClassLoader.java:366)
00:44:17,996 WARN  - at
java.net.URLClassLoader$1.run(URLClassLoader.java:355)
00:44:17,996 WARN  - at
java.security.AccessController.doPrivileged(Native Method)
00:44:17,996 WARN  - at
java.net.URLClassLoader.findClass(URLClassLoader.java:354)
00:44:17,996 WARN  - at
java.lang.ClassLoader.loadClass(ClassLoader.java:425)
00:44:17,996 WARN  - at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
00:44:17,996 WARN  - at
java.lang.ClassLoader.loadClass(ClassLoader.java:358)
00:44:17,996 WARN  - at java.lang.Class.forName0(Native Method)
00:44:17,996 WARN  - at java.lang.Class.forName(Class.java:278)
00:44:17,996 WARN  - at
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:67)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
00:44:17,996 WARN  - at
org.apache.spark.ThrowableSerializationWrapper.readObject(TaskEndReason.scala:167)
00:44:17,996 WARN  - at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
00:44:17,996 WARN  - at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
00:44:17,996 WARN  - at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
00:44:17,996 WARN  - at java.lang.reflect.Method.invoke(Method.java:606)
00:44:17,996 WARN  - at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
00:44:17,996 WARN  - at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
00:44:17,996 WARN  - at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:72)
00:44:17,996 WARN  - at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:98)
00:44:17,996 WARN  - at
org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$2.apply$mcV$sp(TaskResultGetter.scala:108)
00:44:17,996 WARN  - at

Re: Avro vs Protocol buffer for Kafka message

[Selina Tech]

Hi, all:


I like to update some information about Avro message on Kafka. Avro
message include schema ID instead of Schema at each message

http://stackoverflow.com/questions/31204201/apache-kafka-with-avro-and-schema-repo-where-in-the-message-does-the-schema-id

sincerely,
Selina

On Wed, Nov 18, 2015 at 3:49 PM, Selina Tech <swucaree...@gmail.com> wrote:

> Dear All:
>
>   I need to generate some data by Samza to Kafka and then write to
> Parquet formate file.  I was asked why I choose Avro type as my Samza
> output to Kafka instead of Protocol Buffer. Since currently our data on
> Kafka are all Protocol buffer type message.
>
>   I explained that Avro encoded message has advantages such as, the
> encoded size smaller, no extra code compile, implementation easier.  fast
> to serialize/deserialize and supporting a lot language.  However some
> people believe when encoded the Avro message take as much space as Protocol
> buffer, but with schema, the size could be much bigger.
>
>   I am wondering if there are any other advantages make you choose
> Avro as your message type at Kafka? How you consider the data size for Avro
> vs Protocol buffer?
>
> Sincerely,
> Selina
>

Avro vs Protocol buffer for Kafka message

[Selina Tech]

Dear All:

  I need to generate some data by Samza to Kafka and then write to
Parquet formate file.  I was asked why I choose Avro type as my Samza
output to Kafka instead of Protocol Buffer. Since currently our data on
Kafka are all Protocol buffer type message.

  I explained that Avro encoded message has advantages such as, the
encoded size smaller, no extra code compile, implementation easier.  fast
to serialize/deserialize and supporting a lot language.  However some
people believe when encoded the Avro message take as much space as Protocol
buffer, but with schema, the size could be much bigger.

  I am wondering if there are any other advantages make you choose Avro
as your message type at Kafka? How you consider the data size for Avro vs
Protocol buffer?

Sincerely,
Selina

Re: Getting error while reading message

[Selina Tech]

I got same error when I send message to Kafka. generally it caused by
deserializer
as Hemant mentioned.
You need to check how the data was send to Kafka, and how is your consumer
deserializer defined.
And you need to check for both Key and value.

the data is the topic might be byte[] type

Sincerely,
Selina

On Sun, Oct 18, 2015 at 3:01 AM, Kiran Singh  wrote:

> Hi
>
> I am trying to get message from kafka server using hig level consumer API.
> But i am getting following exception
>
> "java.lang.ClassCastException: [B cannot be cast to java.lang.String"
>
> Can anyone explain what this means.
>
> Thanks
> Kiran Singh
>

Re: Kafka - Rest api query

[Selina Tech]

Could you write a consumer at your rest server?

On Tue, Oct 20, 2015 at 1:18 PM, Kudumula, Surender <
surender.kudum...@hpe.com> wrote:

> Thanks for the reply. Iam looking to know if its possible to route binary
> objects messages to rest api service from kafka. If so please let me know.
> Otherwise I can consume the binary object using java consumer and then
> create a rest client and send the binary message via HTTP POST to rest
> server. Appreciate your suggestions. Thanks
>
> Regards
>
> Surender Kudumula
> Big Data Architect - EMEA
> Analytics & Data Management
>
> surender.kudum...@hp.com
> M +44 7795970923
>
> Hewlett-Packard Company
> 88 Wood St
> London EC2V 7QT
> UK
>
>
>
> -Original Message-
> From: Sharninder [mailto:sharnin...@gmail.com]
> Sent: 20 October 2015 16:35
> To: users@kafka.apache.org
> Subject: Re: Kafka - Rest api query
>
> Sounds like an app design decision. What help can this list give you ?
>
>
> > On 20-Oct-2015, at 8:07 PM, Kudumula, Surender <
> surender.kudum...@hpe.com> wrote:
> >
> > Dear sir/madam
> > I have a query. We are working on POC at the moment and we are using
> kafka to produce and consume messages. I have one component which consumes
> the request from topic and processes it and creates a file and again
> produce the java object as byte array to another kafka topic. Now I have
> two options
> > 1. Is to write a java rest client and send the bytearray java object
> to rest api service in another component
> > 2. Or I was wondering if its possible to route the message from
> kafka topic to another component rest APi server.
> >
> > Please let me know we are using kafka 0.8 and new producer. Any
> suggestions would be appreciated?
> >
> > Regards
> >
> > Surender Kudumula
> > Big Data Consultant - EMEA
> > Analytics & Data Management
> >
> > surender.kudum...@hpe.com
> > M +44 7795970923
> >
> > Hewlett-Packard Enterprise
> > Cain Rd,
> > Bracknell
> > RG12 1HN
> > UK
> >
> >
> >
>