[
https://issues.apache.org/jira/browse/SPARK-26267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721869#comment-16721869
]
ASF GitHub Bot commented on SPARK-26267:
----------------------------------------
zsxwing opened a new pull request #23324: [SPARK-26267][SS]Retry when detecting
incorrect offsets from Kafka
URL: https://github.com/apache/spark/pull/23324
## What changes were proposed in this pull request?
Due to [KAFKA-7703](https://issues.apache.org/jira/browse/KAFKA-7703), Kafka
may return an earliest offset when we are request a latest offset. This will
cause Spark to reprocess data.
To reduce the impact of KAFKA-7703, this PR will use the previous offsets we
get to audit the result from Kafka. If we find any incorrect offset, we will
retry at most `maxOffsetFetchAttempts` times. For the first batch of a new
query, as we don't have any previous offsets, we simply fetch offsets twice.
This should reduce the chance to hit KAFKA-7703 a lot.
## How was this patch tested?
Jenkins
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Kafka source may reprocess data
> -------------------------------
>
> Key: SPARK-26267
> URL: https://issues.apache.org/jira/browse/SPARK-26267
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Shixiong Zhu
> Assignee: Shixiong Zhu
> Priority: Blocker
> Labels: correctness
>
> Due to KAFKA-7703, when the Kafka source tries to get the latest offset, it
> may get an earliest offset, and then it will reprocess messages that have
> been processed when it gets the correct latest offset in the next batch.
> This usually happens when restarting a streaming query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
