So, I am just consuming from already existing Kafka queue and topics. According to our internal documentation, when we put event data into Kafka queue, each message has
1 magic byte 4 bytes of Schema ID Then Avro serialized data And we have our Schema Registry server running. Sungwook On Thu, Jun 23, 2016 at 12:25 PM, Tauzell, Dave < dave.tauz...@surescripts.com> wrote: > How are you putting data onto the Topic? The HdfsSink expects that you > used the KafkaAvroSerializer ( > http://docs.confluent.io/1.0/schema-registry/docs/serializer-formatter.html) > which prepends a null byte and schema registry id to the front of the > serialized avro data. If you just put avro onto the topic you will see > this error. > > -Dave > > Dave Tauzell | Senior Software Engineer | Surescripts > O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com > Connect with us: Twitter I LinkedIn I Facebook I YouTube > > > -----Original Message----- > From: Sungwook Yoon [mailto:sy...@maprtech.com] > Sent: Thursday, June 23, 2016 1:46 PM > To: users@kafka.apache.org > Subject: Failed to deserialize data to Avro: > > Hi, > > I am testing kafka connect and got this error, > > Exception in thread "WorkerSinkTask-local-file-sink-0" > org.apache.kafka.connect.errors.DataException: Failed to deserialize data > to Avro: > at > > io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109) > at > > org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266) > at > > org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175) > at > > org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90) > at > > org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58) > at > > org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82) > Caused by: org.apache.kafka.common.errors.SerializationException: Error > deserializing Avro message for id -1 Caused by: > org.apache.kafka.common.errors.SerializationException: Unknown magic byte! > > > I am using > Kafka Connect 2.0.0 > Kafka 0.9 > with Schema Registry Running for Avro Schema Serving. > > Why am I getting this error? > I believe the schema registry server is running fine, since I checked its > running through other means. > > > Any insight would be helpful. > > Thanks, > > Sungwook > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. >
