>> fetch any messages. So, check you have specified the topic name correctly.
>>
>> ~Muthu
>>
>> From: Mail.com [pradeep.mi...@mail.com]
>> Sent: Monday, May 16, 2016 9:33 PM
>> To: Ramaswamy, Muthuraman
>> Cc: Cod
> Cc: Cody Koeninger; spark users
> Subject: Re: KafkaUtils.createDirectStream Not Fetching Messages with
> Confluent Serializers as Value Decoder.
>
> Hi Muthu,
>
> Are you on spark 1.4.1 and Kafka 0.8.2? I have a similar issue even for
> simple string messages.
>
> Consol
From: Mail.com [pradeep.mi...@mail.com]
Sent: Monday, May 16, 2016 9:33 PM
To: Ramaswamy, Muthuraman
Cc: Cody Koeninger; spark users
Subject: Re: KafkaUtils.createDirectStream Not Fetching Messages with Confluent
Serializers as Value Decoder.
Hi Muthu,
Are you on spark 1.4.1 and Kafka 0.8.2? I h
Adding back users.
> On May 18, 2016, at 11:49 AM, Mail.com wrote:
>
> Hi Uladzimir,
>
> I run is as below.
>
> Spark-submit --class com.test --num-executors 4 --executor-cores 5 --queue
> Dev --master yarn-client --driver-memory 512M --executor-memory 512M test.jar
>
> Thanks,
> Pradeep
>
..@insidin.com>"
mailto:j...@insidin.com>>
Date: Tuesday, May 17, 2016 at 3:18 AM
To: "Ramaswamy, Muthuraman"
mailto:muthuraman.ramasw...@viasat.com>>
Cc: spark users mailto:user@spark.apache.org>>
Subject: Re: KafkaUtils.createDirectStream Not Fetching Messages with
I think that if the Confluent deserializer cannot fetch the schema for the
avro message (key and/or value), you end up with no data. You should check
the logs of the Schemaregistry, it should show the HTTP requests it
receives so you can check if the deserializer can connect to it and if so,
what t
Hi Muthu,
Are you on spark 1.4.1 and Kafka 0.8.2? I have a similar issue even for simple
string messages.
Console producer and consumer work fine. But spark always reruns empty RDD. I
am using Receiver based Approach.
Thanks,
Pradeep
> On May 16, 2016, at 8:19 PM, Ramaswamy, Muthuraman
> w
Yes, I can see the messages. Also, I wrote a quick custom decoder for avro and
it works fine for the following:
>> kvs = KafkaUtils.createDirectStream(ssc, [topic], {"metadata.broker.list":
>> brokers}, valueDecoder=decoder)
But, when I use the Confluent Serializers to leverage the Schema Regis
Have you checked to make sure you can receive messages just using a
byte array for value?
On Mon, May 16, 2016 at 12:33 PM, Ramaswamy, Muthuraman
wrote:
> I am trying to consume AVRO formatted message through
> KafkaUtils.createDirectStream. I followed the listed below example (refer
> link) but
