Re: Error "Unknown magic byte" occurred while deserializing Avro message

2016-11-26 Thread Ewen Cheslack-Postava 
Are you sure you have not produced any other data into that topic, e.g.
perhaps you were testing the regular kafka-console-producer before? This
would cause it to fail on the non-Avro messages (as Dayong says, because
the initial magic byte mismatches).

Can you try starting the consumer first without --from-beginning and then
running the producer to add more data to the topic? If that works, you can
dump the raw bytes using kafka-console-consumer to try to figure out where
the bad data came from, but I'd guess since it looks like you're just
testing you might also just want to delete the topic to get it back to a
clean state.

-Ewen

On Wed, Nov 23, 2016 at 4:27 AM, Dayong <will...@gmail.com> wrote:

> As I remember this is to complain the fist byte of msg is not x00. I think
> console producer does not support json since it uses string schema.
>
> Thanks,
> Dayong
>
> > On Nov 23, 2016, at 4:28 AM, ZHU Hua B <hua.b@alcatel-lucent.com>
> wrote:
> >
> > Hi,
> >
> >
> > We tried to produce and consume a AVRO message (zookeeper, broker and
> schema registry have been launched), error "Unknown magic byte" occurred
> while deserializing Avro message, if I missed anything? Thanks!
> >
> >
> > From producer:
> > # bin/kafka-avro-console-producer --broker-list localhost:9092 --topic
> test --property value.schema='{"type":"record"
> ,"name":"myrecord","fields":[{"name":"f1","type":"string"}]}'
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in [jar:file:/root/confluent/shar
> e/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/
> impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in [jar:file:/root/confluent/shar
> e/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/
> impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in [jar:file:/root/confluent/shar
> e/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/
> impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >
> > {"f1": "value1"}
> >
> >
> > From consumer:
> > # bin/kafka-avro-console-consumer --topic test --zookeeper
> localhost:2181 --from-beginning
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in [jar:file:/root/confluent/shar
> e/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/
> impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in [jar:file:/root/confluent/shar
> e/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/
> impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in [jar:file:/root/confluent/shar
> e/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/
> impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >
> > [2016-11-23 16:15:11,108] ERROR Unknown error when running consumer:
> (kafka.tools.ConsoleConsumer$:103)
> > org.apache.kafka.common.errors.SerializationException: Error
> deserializing Avro message for id -1
> > Caused by: org.apache.kafka.common.errors.SerializationException:
> Unknown magic byte!
> > [2016-11-23 16:15:11,108] ERROR Unknown error when running consumer:
> (kafka.tools.ConsoleConsumer$:103)
> > org.apache.kafka.common.errors.SerializationException: Error
> deserializing Avro message for id -1
> > Caused by: org.apache.kafka.common.errors.SerializationException:
> Unknown magic byte!
> >
> >
> > Best Regards
> >
> > Johnny
> >
>



-- 
Thanks,
Ewen

Re: Error "Unknown magic byte" occurred while deserializing Avro message

2016-11-23 Thread Dayong 
As I remember this is to complain the fist byte of msg is not x00. I think 
console producer does not support json since it uses string schema. 

Thanks,
Dayong

> On Nov 23, 2016, at 4:28 AM, ZHU Hua B <hua.b@alcatel-lucent.com> wrote:
> 
> Hi,
> 
> 
> We tried to produce and consume a AVRO message (zookeeper, broker and schema 
> registry have been launched), error "Unknown magic byte" occurred while 
> deserializing Avro message, if I missed anything? Thanks!
> 
> 
> From producer:
> # bin/kafka-avro-console-producer --broker-list localhost:9092 --topic test 
> --property 
> value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}'
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/root/confluent/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/root/confluent/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/root/confluent/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 
> {"f1": "value1"}
> 
> 
> From consumer:
> # bin/kafka-avro-console-consumer --topic test --zookeeper localhost:2181 
> --from-beginning
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/root/confluent/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/root/confluent/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/root/confluent/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 
> [2016-11-23 16:15:11,108] ERROR Unknown error when running consumer:  
> (kafka.tools.ConsoleConsumer$:103)
> org.apache.kafka.common.errors.SerializationException: Error deserializing 
> Avro message for id -1
> Caused by: org.apache.kafka.common.errors.SerializationException: Unknown 
> magic byte!
> [2016-11-23 16:15:11,108] ERROR Unknown error when running consumer:  
> (kafka.tools.ConsoleConsumer$:103)
> org.apache.kafka.common.errors.SerializationException: Error deserializing 
> Avro message for id -1
> Caused by: org.apache.kafka.common.errors.SerializationException: Unknown 
> magic byte!
> 
> 
> Best Regards
> 
> Johnny
>

Error "Unknown magic byte" occurred while deserializing Avro message

2016-11-23 Thread ZHU Hua B 
Hi,


We tried to produce and consume a AVRO message (zookeeper, broker and schema 
registry have been launched), error "Unknown magic byte" occurred while 
deserializing Avro message, if I missed anything? Thanks!


>From producer:
# bin/kafka-avro-console-producer --broker-list localhost:9092 --topic test 
--property 
value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}'
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/root/confluent/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/root/confluent/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/root/confluent/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

{"f1": "value1"}


>From consumer:
# bin/kafka-avro-console-consumer --topic test --zookeeper localhost:2181 
--from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/root/confluent/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/root/confluent/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/root/confluent/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

[2016-11-23 16:15:11,108] ERROR Unknown error when running consumer:  
(kafka.tools.ConsoleConsumer$:103)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro 
message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic 
byte!
[2016-11-23 16:15:11,108] ERROR Unknown error when running consumer:  
(kafka.tools.ConsoleConsumer$:103)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro 
message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic 
byte!


Best Regards

Johnny