RE: Failed to deserialize data to Avro:
That should work then. I would take some messages off the queue and verify that they have the correct magic byte (byte 0). -Dave Dave Tauzell | Senior Software Engineer | Surescripts O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com Connect with us: Twitter I LinkedIn I Facebook I YouTube -Original Message- From: Sungwook Yoon [mailto:sy...@maprtech.com] Sent: Thursday, June 23, 2016 2:32 PM To: users@kafka.apache.org Subject: Re: Failed to deserialize data to Avro: So, I am just consuming from already existing Kafka queue and topics. According to our internal documentation, when we put event data into Kafka queue, each message has 1 magic byte 4 bytes of Schema ID Then Avro serialized data And we have our Schema Registry server running. Sungwook On Thu, Jun 23, 2016 at 12:25 PM, Tauzell, Dave < dave.tauz...@surescripts.com> wrote: > How are you putting data onto the Topic? The HdfsSink expects that > you used the KafkaAvroSerializer ( > http://docs.confluent.io/1.0/schema-registry/docs/serializer-formatter > .html) which prepends a null byte and schema registry id to the front > of the serialized avro data. If you just put avro onto the topic you > will see this error. > > -Dave > > Dave Tauzell | Senior Software Engineer | Surescripts > O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com > Connect with us: Twitter I LinkedIn I Facebook I YouTube > > > -Original Message- > From: Sungwook Yoon [mailto:sy...@maprtech.com] > Sent: Thursday, June 23, 2016 1:46 PM > To: users@kafka.apache.org > Subject: Failed to deserialize data to Avro: > > Hi, > > I am testing kafka connect and got this error, > > Exception in thread "WorkerSinkTask-local-file-sink-0" > org.apache.kafka.connect.errors.DataException: Failed to deserialize > data to Avro: > at > > io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109) > at > > org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266) > at > > org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175) > at > > org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90) > at > > org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58) > at > > org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThrea > d.java:82) Caused by: > org.apache.kafka.common.errors.SerializationException: Error > deserializing Avro message for id -1 Caused by: > org.apache.kafka.common.errors.SerializationException: Unknown magic byte! > > > I am using > Kafka Connect 2.0.0 > Kafka 0.9 > with Schema Registry Running for Avro Schema Serving. > > Why am I getting this error? > I believe the schema registry server is running fine, since I checked > its running through other means. > > > Any insight would be helpful. > > Thanks, > > Sungwook > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of > the individual or entity to whom they are addressed. If you have > received this e-mail in error, please notify the sender by reply > e-mail immediately and destroy all copies of the e-mail and any attachments. > This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Failed to deserialize data to Avro:
So, I am just consuming from already existing Kafka queue and topics. According to our internal documentation, when we put event data into Kafka queue, each message has 1 magic byte 4 bytes of Schema ID Then Avro serialized data And we have our Schema Registry server running. Sungwook On Thu, Jun 23, 2016 at 12:25 PM, Tauzell, Dave < dave.tauz...@surescripts.com> wrote: > How are you putting data onto the Topic? The HdfsSink expects that you > used the KafkaAvroSerializer ( > http://docs.confluent.io/1.0/schema-registry/docs/serializer-formatter.html) > which prepends a null byte and schema registry id to the front of the > serialized avro data. If you just put avro onto the topic you will see > this error. > > -Dave > > Dave Tauzell | Senior Software Engineer | Surescripts > O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com > Connect with us: Twitter I LinkedIn I Facebook I YouTube > > > -Original Message- > From: Sungwook Yoon [mailto:sy...@maprtech.com] > Sent: Thursday, June 23, 2016 1:46 PM > To: users@kafka.apache.org > Subject: Failed to deserialize data to Avro: > > Hi, > > I am testing kafka connect and got this error, > > Exception in thread "WorkerSinkTask-local-file-sink-0" > org.apache.kafka.connect.errors.DataException: Failed to deserialize data > to Avro: > at > > io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109) > at > > org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266) > at > > org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175) > at > > org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90) > at > > org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58) > at > > org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82) > Caused by: org.apache.kafka.common.errors.SerializationException: Error > deserializing Avro message for id -1 Caused by: > org.apache.kafka.common.errors.SerializationException: Unknown magic byte! > > > I am using > Kafka Connect 2.0.0 > Kafka 0.9 > with Schema Registry Running for Avro Schema Serving. > > Why am I getting this error? > I believe the schema registry server is running fine, since I checked its > running through other means. > > > Any insight would be helpful. > > Thanks, > > Sungwook > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. >
RE: Failed to deserialize data to Avro:
How are you putting data onto the Topic? The HdfsSink expects that you used the KafkaAvroSerializer (http://docs.confluent.io/1.0/schema-registry/docs/serializer-formatter.html) which prepends a null byte and schema registry id to the front of the serialized avro data. If you just put avro onto the topic you will see this error. -Dave Dave Tauzell | Senior Software Engineer | Surescripts O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com Connect with us: Twitter I LinkedIn I Facebook I YouTube -Original Message- From: Sungwook Yoon [mailto:sy...@maprtech.com] Sent: Thursday, June 23, 2016 1:46 PM To: users@kafka.apache.org Subject: Failed to deserialize data to Avro: Hi, I am testing kafka connect and got this error, Exception in thread "WorkerSinkTask-local-file-sink-0" org.apache.kafka.connect.errors.DataException: Failed to deserialize data to Avro: at io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109) at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175) at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90) at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58) at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82) Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1 Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte! I am using Kafka Connect 2.0.0 Kafka 0.9 with Schema Registry Running for Avro Schema Serving. Why am I getting this error? I believe the schema registry server is running fine, since I checked its running through other means. Any insight would be helpful. Thanks, Sungwook This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
