:01:01 PM
To: Bryan Jeffrey
Cc: user
Subject: Re: Kafka Offset Storage: Fetching Offsets
Offsets are loaded when you instantiate an
org.apache.kafka.clients.consumer.KafkaConsumer, subscribe, and poll.
There's not an explicit api for it. Have you looked at the output of
kafka-consumer-gro
2018 4:24:58 PM
>
> To: Bryan Jeffrey
> Cc: user
> Subject: Re: Kafka Offset Storage: Fetching Offsets
>
> The code that loads offsets from kafka is in e.g.
> org.apache.kafka.clients.consumer, it's not in spark.
>
> On Thu, Jun 14, 2018 at 3:22 PM, Bryan Jeffrey
>
ubject: Re: Kafka Offset Storage: Fetching Offsets
The code that loads offsets from kafka is in e.g.
org.apache.kafka.clients.consumer, it's not in spark.
On Thu, Jun 14, 2018 at 3:22 PM, Bryan Jeffrey wrote:
> Cody,
>
> Can you point me to the code that loads offsets? As far as I can s
m checkpoint.
>
> Thank you!
>
> Bryan
>
> Get Outlook for Android
>
>
> From: Cody Koeninger
> Sent: Thursday, June 14, 2018 4:00:31 PM
> To: Bryan Jeffrey
> Cc: user
> Subject: Re: Kafka Offset Storage: Fetching Off
:31 PM
To: Bryan Jeffrey
Cc: user
Subject: Re: Kafka Offset Storage: Fetching Offsets
The expectation is that you shouldn't have to manually load offsets
from kafka, because the underlying kafka consumer on the driver will
start at the offsets associated with the given group id.
That's the behavior
The expectation is that you shouldn't have to manually load offsets
from kafka, because the underlying kafka consumer on the driver will
start at the offsets associated with the given group id.
That's the behavior I see with this example:
Hello.
I am using Spark 2.1 and Kafka 0.10.2.1 and the DStream interface. Based
on the documentation (
https://spark.apache.org/docs/2.1.0/streaming-kafka-0-10-integration.html#kafka-itself),
it appears that you can now use Kafka itself to store offsets.
I've setup a simple Kafka DStream:
val
