[streaming] reading Kafka direct stream throws kafka.common.OffsetOutOfRangeException

[Alexey Ponkin]

Hi

I have simple spark-streaming job(8 executors 1 core - on 8 node cluster) - 
read from Kafka topic( 3 brokers with 8 partitions) and save to Cassandra.
The problem is that when I increase number of incoming messages in topic the 
job is starting to fail with kafka.common.OffsetOutOfRangeException.
Job fails starting from 100 events per second.

Thanks in advance

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: [streaming] reading Kafka direct stream throws kafka.common.OffsetOutOfRangeException

[Cody Koeninger]

Offset out of range means the message in question is no longer available on
Kafka.  What's your kafka log retention set to, and how does that compare
to your processing time?

On Wed, Sep 30, 2015 at 4:26 AM, Alexey Ponkin  wrote:

> Hi
>
> I have simple spark-streaming job(8 executors 1 core - on 8 node cluster)
> - read from Kafka topic( 3 brokers with 8 partitions) and save to Cassandra.
> The problem is that when I increase number of incoming messages in topic
> the job is starting to fail with kafka.common.OffsetOutOfRangeException.
> Job fails starting from 100 events per second.
>
> Thanks in advance
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>