[
https://issues.apache.org/jira/browse/SPARK-24720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quentin Ambard updated SPARK-24720:
-----------------------------------
Description:
When kafka transactions are used, sending 1 message to kafka will result to 1
offset for the data + 1 offset to mark the transaction.
When kafka connector for spark streaming read a topic with non-consecutive
offset, it leads to a failure. SPARK-17147 fixed this issue for compacted
topics.
However, SPARK-17147 doesn't fix this issue: if 1 message + 1 transaction
commit are in a partition, spark will try to read offsets [0 2[. offset 0
(containing the message) will be read, but offset 1 won't return a value and
buffer.hasNext() will be false even after a poll since no data are present for
offset 1 (it's the transaction commit)
was:
When kafka transactions are used, sending 1 message to kafka will result to 1
offset for the data + 1 offset to mark the transaction.
When kafka read a topic with non-consecutive offset, it leads to a failure.
SPARK-17147 fixed this issue for compacted topics.
However, SPARK-17147 doesn't fix this issue: if 1 message + 1 transaction
commit are in a partition, spark will try to read offsets [0 2[. offset 0
(containing the message) will be read, but offset 1 won't return a value and
buffer.hasNext() will be false even after a poll since no data are present for
offset 1 (it's the transaction commit)
> kafka transaction creates Non-consecutive Offsets (due to transaction offset)
> making streaming fail when failOnDataLoss=true
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-24720
> URL: https://issues.apache.org/jira/browse/SPARK-24720
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.3.1
> Reporter: Quentin Ambard
> Priority: Major
>
> When kafka transactions are used, sending 1 message to kafka will result to 1
> offset for the data + 1 offset to mark the transaction.
> When kafka connector for spark streaming read a topic with non-consecutive
> offset, it leads to a failure. SPARK-17147 fixed this issue for compacted
> topics.
> However, SPARK-17147 doesn't fix this issue: if 1 message + 1 transaction
> commit are in a partition, spark will try to read offsets [0 2[. offset 0
> (containing the message) will be read, but offset 1 won't return a value and
> buffer.hasNext() will be false even after a poll since no data are present
> for offset 1 (it's the transaction commit)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
