Re: org.apache.kafka.clients.consumer.OffsetOutOfRangeException

[dcam]

Hi Mina

I believe this is different for Structured Streaming from Kafka,
specifically. I'm assuming you are using structured streaming based on the
name of the dependency ("spark-streaming-kafka"). There is a note in the
docs here:
https://spark.apache.org/docs/2.2.0/structured-streaming-kafka-integration.html#kafka-specific-configurations

So, instead of setting Kafka's client property, instead set the Spark config
"startingOffsets" to "earliest". There are examples here:
https://spark.apache.org/docs/2.2.0/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries

Setting "startingOffsets" has worked for me. I have not tried setting the
kafka property directly.

Cheers,
Dave



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

org.apache.kafka.clients.consumer.OffsetOutOfRangeException

[Mina Aslani]

Hi,

I am getting below error
Caused by: org.apache.kafka.clients.consumer.OffsetOutOfRangeException:
Offsets out of range with no configured reset policy for partitions:
{topic1-0=304337}
as soon as I submit a spark app to my cluster.

I am using below dependency
name: 'spark-streaming-kafka-0-10_2.11', version: '2.2.0' And setting the
consumer's reset config(e.g. AUTO_OFFSET_RESET_CONFIG) to "earliest".
As per https://kafka.apache.org/0110/documentation.html the exception
should be thrown only when the consumer's reset config has not been set
(e.g. default=none).
Wondering what is the cause and how to fix.

Best regards,
Mina