Re: Spark Streaming fails with unable to get records after polling for 512 ms

I don't see anything obvious, you'd need to do more troubleshooting. Could also try creating a single rdd for a known range of offsets: http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html#creating-an-rdd On Wed, Nov 15, 2017 at 9:33 PM, jagadish kagitala

Re: Spark Streaming fails with unable to get records after polling for 512 ms

Hi Cody, It worked, after moving the parameter to sparkConf. I don't see that error. But, Now i'm seeing the count for each RDD returns 0. But, there are records in the topic i'm reading. Do you see anything wrong with how i'm creating the Direct Stream ? Thanks Jagadish On Wed, Nov 15, 2017

Re: Spark Streaming fails with unable to get records after polling for 512 ms

spark.streaming.kafka.consumer.poll.ms is a spark configuration, not a kafka parameter. see http://spark.apache.org/docs/latest/configuration.html On Tue, Nov 14, 2017 at 8:56 PM, jkagitala wrote: > Hi, > > I'm trying to add spark-streaming to our kafka topic. But, I keep

Spark Streaming fails with unable to get records after polling for 512 ms

Hi, I'm trying to add spark-streaming to our kafka topic. But, I keep getting this error java.lang.AssertionError: assertion failed: Failed to get record after polling for 512 ms. I tried to add different params like max.poll.interval.ms, spark.streaming.kafka.consumer.poll.ms to 1ms in