Re: Issue Storing offset in Kafka for Spark Streaming Application

2017-10-13 Thread Arpan Rajani
Hi Gerard, Excellent, indeed your inputs helped. Thank you for the quick reply. I modified the code based on inputs. Now the application starts and it reads from the topic. Now we stream like 50,000 messages on the Kafka topic. After a while we terminate the application using YARN kill and

Re: Issue Storing offset in Kafka for Spark Streaming Application

2017-10-13 Thread Gerard Maas
Hi Arpan, The error suggests that the streaming context has been started with streamingContext.start() and after that statement, some other dstream operations have been attempted. A suggested pattern to manage the offsets is the following: var offsetRanges: Array[OffsetRanger] = _ //create