So, I am just consuming from already existing Kafka queue and topics.
According to our internal documentation, when we put event data into Kafka
queue, each message has
1 magic byte
4 bytes of Schema ID
Then Avro serialized data
And we have our Schema Registry server running.
Sungwook
On Thu, Jun 23, 2016 at 12:25 PM, Tauzell, Dave <
dave.tauz...@surescripts.com> wrote:
> How are you putting data onto the Topic? The HdfsSink expects that you
> used the KafkaAvroSerializer (
> http://docs.confluent.io/1.0/schema-registry/docs/serializer-formatter.html)
> which prepends a null byte and schema registry id to the front of the
> serialized avro data. If you just put avro onto the topic you will see
> this error.
>
> -Dave
>
> Dave Tauzell | Senior Software Engineer | Surescripts
> O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com
> Connect with us: Twitter I LinkedIn I Facebook I YouTube
>
>
> -Original Message-
> From: Sungwook Yoon [mailto:sy...@maprtech.com]
> Sent: Thursday, June 23, 2016 1:46 PM
> To: users@kafka.apache.org
> Subject: Failed to deserialize data to Avro:
>
> Hi,
>
> I am testing kafka connect and got this error,
>
> Exception in thread "WorkerSinkTask-local-file-sink-0"
> org.apache.kafka.connect.errors.DataException: Failed to deserialize data
> to Avro:
> at
>
> io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:109)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
> at
>
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
> at
>
> org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
> Caused by: org.apache.kafka.common.errors.SerializationException: Error
> deserializing Avro message for id -1 Caused by:
> org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
>
>
> I am using
> Kafka Connect 2.0.0
> Kafka 0.9
> with Schema Registry Running for Avro Schema Serving.
>
> Why am I getting this error?
> I believe the schema registry server is running fine, since I checked its
> running through other means.
>
>
> Any insight would be helpful.
>
> Thanks,
>
> Sungwook
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>