[jira] [Updated] (STORM-2666) Storm-kafka-client spout can sometimes emit messages that were already committed.
[
https://issues.apache.org/jira/browse/STORM-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stig Rohde Døssing updated STORM-2666:
--
Fix Version/s: 2.0.0
> Storm-kafka-client spout can sometimes emit messages that were already
> committed.
> --
>
> Key: STORM-2666
> URL: https://issues.apache.org/jira/browse/STORM-2666
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.1.1, 1.2.0
>Reporter: Guang Du
>Assignee: Stig Rohde Døssing
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Under a certain heavy load, for failed/timeout tuples, the retry service will
> ack tuple for failed max times. Kafka Client Spout will commit after reached
> the commit interval. However seems some 'on the way' tuples will be failed
> again, the retry service will cause Spout to emit again, and acked eventually
> to OffsetManager.
> In some cases such offsets are too many, exceeding the max-uncommit, causing
> org.apache.storm.kafka.spout.internal.OffsetManager#findNextCommitOffset
> unable to find next commit point, and Spout for this partition will not poll
> any more.
> By the way I've applied STORM-2549 PR#2156 from Stig Døssing to fix
> STORM-2625, and I'm using Python Shell Bolt as processing bolt, if this
> information helps.
> resulting logs like below. I'm not sure if the issue has already been
> raised/fixed, glad if anyone could help to point out existing JIRA. Thank you.
> 2017-07-27 22:23:48.398 o.a.s.k.s.KafkaSpout Thread-23-spout-executor[248
> 248] [INFO] Successful ack for tuple message
> [{topic-partition=kafka_bd_trigger_action-20, offset=18204, numFails=0}].
> 2017-07-27 22:23:49.203 o.a.s.k.s.i.OffsetManager
> Thread-23-spout-executor[248 248] [WARN] topic-partition
> [kafka_bd_trigger_action-18] has unexpected offset [16002]. Current committed
> Offset [16003]
> Edit:
> See
> https://issues.apache.org/jira/browse/STORM-2666?focusedCommentId=16125893=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16125893
> for the current best guess at the root cause of this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-2666) Storm-kafka-client spout can sometimes emit messages that were already committed.
[
https://issues.apache.org/jira/browse/STORM-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated STORM-2666:
--
Labels: pull-request-available (was: )
> Storm-kafka-client spout can sometimes emit messages that were already
> committed.
> --
>
> Key: STORM-2666
> URL: https://issues.apache.org/jira/browse/STORM-2666
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.1.1, 1.2.0
>Reporter: Guang Du
>Assignee: Stig Rohde Døssing
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Under a certain heavy load, for failed/timeout tuples, the retry service will
> ack tuple for failed max times. Kafka Client Spout will commit after reached
> the commit interval. However seems some 'on the way' tuples will be failed
> again, the retry service will cause Spout to emit again, and acked eventually
> to OffsetManager.
> In some cases such offsets are too many, exceeding the max-uncommit, causing
> org.apache.storm.kafka.spout.internal.OffsetManager#findNextCommitOffset
> unable to find next commit point, and Spout for this partition will not poll
> any more.
> By the way I've applied STORM-2549 PR#2156 from Stig Døssing to fix
> STORM-2625, and I'm using Python Shell Bolt as processing bolt, if this
> information helps.
> resulting logs like below. I'm not sure if the issue has already been
> raised/fixed, glad if anyone could help to point out existing JIRA. Thank you.
> 2017-07-27 22:23:48.398 o.a.s.k.s.KafkaSpout Thread-23-spout-executor[248
> 248] [INFO] Successful ack for tuple message
> [{topic-partition=kafka_bd_trigger_action-20, offset=18204, numFails=0}].
> 2017-07-27 22:23:49.203 o.a.s.k.s.i.OffsetManager
> Thread-23-spout-executor[248 248] [WARN] topic-partition
> [kafka_bd_trigger_action-18] has unexpected offset [16002]. Current committed
> Offset [16003]
> Edit:
> See
> https://issues.apache.org/jira/browse/STORM-2666?focusedCommentId=16125893=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16125893
> for the current best guess at the root cause of this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-2666) Storm-kafka-client spout can sometimes emit messages that were already committed.
[
https://issues.apache.org/jira/browse/STORM-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stig Rohde Døssing updated STORM-2666:
--
Summary: Storm-kafka-client spout can sometimes emit messages that were
already committed. (was: Kafka Client Spout send & ack committed offsets)
> Storm-kafka-client spout can sometimes emit messages that were already
> committed.
> --
>
> Key: STORM-2666
> URL: https://issues.apache.org/jira/browse/STORM-2666
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.1.1, 1.2.0
>Reporter: Guang Du
>Assignee: Stig Rohde Døssing
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Under a certain heavy load, for failed/timeout tuples, the retry service will
> ack tuple for failed max times. Kafka Client Spout will commit after reached
> the commit interval. However seems some 'on the way' tuples will be failed
> again, the retry service will cause Spout to emit again, and acked eventually
> to OffsetManager.
> In some cases such offsets are too many, exceeding the max-uncommit, causing
> org.apache.storm.kafka.spout.internal.OffsetManager#findNextCommitOffset
> unable to find next commit point, and Spout for this partition will not poll
> any more.
> By the way I've applied STORM-2549 PR#2156 from Stig Døssing to fix
> STORM-2625, and I'm using Python Shell Bolt as processing bolt, if this
> information helps.
> resulting logs like below. I'm not sure if the issue has already been
> raised/fixed, glad if anyone could help to point out existing JIRA. Thank you.
> 2017-07-27 22:23:48.398 o.a.s.k.s.KafkaSpout Thread-23-spout-executor[248
> 248] [INFO] Successful ack for tuple message
> [{topic-partition=kafka_bd_trigger_action-20, offset=18204, numFails=0}].
> 2017-07-27 22:23:49.203 o.a.s.k.s.i.OffsetManager
> Thread-23-spout-executor[248 248] [WARN] topic-partition
> [kafka_bd_trigger_action-18] has unexpected offset [16002]. Current committed
> Offset [16003]
> Edit:
> See
> https://issues.apache.org/jira/browse/STORM-2666?focusedCommentId=16125893=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16125893
> for the current best guess at the root cause of this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
