Thank you for the input. Apparently, I was referring to incorrect Schema Registry Server. Once the correct Schema Registry Server IP is used, serializer worked for me.
Thanks again, ~Muthu From: Jan Uyttenhove <j...@insidin.com<mailto:j...@insidin.com>> Reply-To: "j...@insidin.com<mailto:j...@insidin.com>" <j...@insidin.com<mailto:j...@insidin.com>> Date: Tuesday, May 17, 2016 at 3:18 AM To: "Ramaswamy, Muthuraman" <muthuraman.ramasw...@viasat.com<mailto:muthuraman.ramasw...@viasat.com>> Cc: spark users <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: KafkaUtils.createDirectStream Not Fetching Messages with Confluent Serializers as Value Decoder. I think that if the Confluent deserializer cannot fetch the schema for the avro message (key and/or value), you end up with no data. You should check the logs of the Schemaregistry, it should show the HTTP requests it receives so you can check if the deserializer can connect to it and if so, what the response code looks like. If you use the Confluent serializer, each avro message is first serialized and afterwards the schema id is added to it. This way, the Confluent deserializer can fetch the schema id first and use it to lookup the schema in the Schemaregistry. On Tue, May 17, 2016 at 2:19 AM, Ramaswamy, Muthuraman <muthuraman.ramasw...@viasat.com<mailto:muthuraman.ramasw...@viasat.com>> wrote: Yes, I can see the messages. Also, I wrote a quick custom decoder for avro and it works fine for the following: >> kvs = KafkaUtils.createDirectStream(ssc, [topic], {"metadata.broker.list": >> brokers}, valueDecoder=decoder) But, when I use the Confluent Serializers to leverage the Schema Registry (based on the link shown below), it doesn’t work for me. I am not sure whether I need to configure any more details to consume the Schema Registry. I can fetch the schema from the schema registry based on is Ids. The decoder method is not returning any values for me. ~Muthu On 5/16/16, 10:49 AM, "Cody Koeninger" <c...@koeninger.org<mailto:c...@koeninger.org>> wrote: >Have you checked to make sure you can receive messages just using a >byte array for value? > >On Mon, May 16, 2016 at 12:33 PM, Ramaswamy, Muthuraman ><muthuraman.ramasw...@viasat.com<mailto:muthuraman.ramasw...@viasat.com>> >wrote: >> I am trying to consume AVRO formatted message through >> KafkaUtils.createDirectStream. I followed the listed below example (refer >> link) but the messages are not being fetched by the Stream. >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__stackoverflow.com_questions_30339636_spark-2Dpython-2Davro-2Dkafka-2Ddeserialiser&d=CwIBaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=NQ-dw5X8CJcqaXIvIdMUUdkL0fHDonD07FZzTY3CgiU&m=Nc-rPMFydyCrwOZuNWs2GmSL4NkN8eGoR-mkJUlkCx0&s=hwqxCKl3P4_9pKWeo1OGR134QegMRe3Xh22_WMy-5q8&e= >> >> Is there any code missing that I must add to make the above sample work. >> Say, I am not sure how the confluent serializers would know the avro schema >> info as it knows only the Schema Registry URL info. >> >> Appreciate your help. >> >> ~Muthu >> >> >> -- Jan Uyttenhove Streaming data & digital solutions architect @ Insidin bvba j...@insidin.com<mailto:j...@insidin.com> +32 474 56 24 39 https://twitter.com/xorto<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_xorto&d=CwMFaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=NQ-dw5X8CJcqaXIvIdMUUdkL0fHDonD07FZzTY3CgiU&m=NHdl0YMOAR5ds_GMw-e5RQ4GX7_2pz0JGCveDNHYpcg&s=IgEUqhGm729eTbbVDC3X1RqxdU286XK5-V-21ir25VM&e=> https://www.linkedin.com/in/januyttenhove<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_in_januyttenhove&d=CwMFaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=NQ-dw5X8CJcqaXIvIdMUUdkL0fHDonD07FZzTY3CgiU&m=NHdl0YMOAR5ds_GMw-e5RQ4GX7_2pz0JGCveDNHYpcg&s=dPIFPsJrLbWkKrnnLpOhcFZ9zHgBN1LLRC0SKQGFz6U&e=> This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed. It may contain privileged and confidential information. If you are not the intended recipient please notify the sender immediately and destroy this e-mail. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication of this e-mail message is strictly prohibited. Whilst all efforts are made to safeguard e-mails, the sender cannot guarantee that attachments are virus free or compatible with your systems and does not accept liability in respect of viruses or computer problems experienced.
