I think that if the Confluent deserializer cannot fetch the schema for the
avro message (key and/or value), you end up with no data. You should check
the logs of the Schemaregistry, it should show the HTTP requests it
receives so you can check if the deserializer can connect to it and if so,
what the response code looks like.
If you use the Confluent serializer, each avro message is first serialized
and afterwards the schema id is added to it. This way, the Confluent
deserializer can fetch the schema id first and use it to lookup the schema
in the Schemaregistry.
On Tue, May 17, 2016 at 2:19 AM, Ramaswamy, Muthuraman <
muthuraman.ramasw...@viasat.com> wrote:
> Yes, I can see the messages. Also, I wrote a quick custom decoder for avro
> and it works fine for the following:
>
> >> kvs = KafkaUtils.createDirectStream(ssc, [topic],
> {"metadata.broker.list": brokers}, valueDecoder=decoder)
>
> But, when I use the Confluent Serializers to leverage the Schema Registry
> (based on the link shown below), it doesn’t work for me. I am not sure
> whether I need to configure any more details to consume the Schema
> Registry. I can fetch the schema from the schema registry based on is Ids.
> The decoder method is not returning any values for me.
>
> ~Muthu
>
>
>
> On 5/16/16, 10:49 AM, "Cody Koeninger" <c...@koeninger.org> wrote:
>
> >Have you checked to make sure you can receive messages just using a
> >byte array for value?
> >
> >On Mon, May 16, 2016 at 12:33 PM, Ramaswamy, Muthuraman
> ><muthuraman.ramasw...@viasat.com> wrote:
> >> I am trying to consume AVRO formatted message through
> >> KafkaUtils.createDirectStream. I followed the listed below example
> (refer
> >> link) but the messages are not being fetched by the Stream.
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__stackoverflow.com_questions_30339636_spark-2Dpython-2Davro-2Dkafka-2Ddeserialiser&d=CwIBaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=NQ-dw5X8CJcqaXIvIdMUUdkL0fHDonD07FZzTY3CgiU&m=Nc-rPMFydyCrwOZuNWs2GmSL4NkN8eGoR-mkJUlkCx0&s=hwqxCKl3P4_9pKWeo1OGR134QegMRe3Xh22_WMy-5q8&e=
> >>
> >> Is there any code missing that I must add to make the above sample work.
> >> Say, I am not sure how the confluent serializers would know the avro
> schema
> >> info as it knows only the Schema Registry URL info.
> >>
> >> Appreciate your help.
> >>
> >> ~Muthu
> >>
> >>
> >>
>
--
Jan Uyttenhove
Streaming data & digital solutions architect @ Insidin bvba
j...@insidin.com
+32 474 56 24 39
https://twitter.com/xorto
https://www.linkedin.com/in/januyttenhove
This e-mail and any files transmitted with it are intended solely for the
use of the individual or entity to whom they are addressed. It may contain
privileged and confidential information. If you are not the intended
recipient please notify the sender immediately and destroy this e-mail. Any
form of reproduction, dissemination, copying, disclosure, modification,
distribution and/or publication of this e-mail message is strictly
prohibited. Whilst all efforts are made to safeguard e-mails, the sender
cannot guarantee that attachments are virus free or compatible with your
systems and does not accept liability in respect of viruses or computer
problems experienced.
