[jira] [Updated] (STORM-2546) Kafka spout can stall / get stuck due to edge case with failing tuples
[ https://issues.apache.org/jira/browse/STORM-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated STORM-2546: -- Labels: pull-request-available (was: ) > Kafka spout can stall / get stuck due to edge case with failing tuples > -- > > Key: STORM-2546 > URL: https://issues.apache.org/jira/browse/STORM-2546 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.x >Reporter: Prasanna Ranganathan >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The mechanism for replaying a failed tuple involves seeking the kafka > consumer to the failing offset and then re-emitting it into the topology. A > tuple, when emitted the first time, will have an entry created in > OffsetManager. This entry will be removed only after the tuple is > successfully acknowledged and its offset successfully committed. Till then, > commits for offsets beyond the failing offset for that TopicPartition will be > blocked. > It is possible that when the spout seeks the consumer to the failing offset, > the corresponding kafka message is not returned in the poll response. This > can happen due to that offset being deleted or compacted away. In this > scenario that partition will be blocked from committing and progressing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2546) Kafka spout can stall / get stuck due to edge case with failing tuples
[ https://issues.apache.org/jira/browse/STORM-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2546: -- Affects Version/s: (was: 1.x) 1.1.0 > Kafka spout can stall / get stuck due to edge case with failing tuples > -- > > Key: STORM-2546 > URL: https://issues.apache.org/jira/browse/STORM-2546 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0 >Reporter: Prasanna Ranganathan >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The mechanism for replaying a failed tuple involves seeking the kafka > consumer to the failing offset and then re-emitting it into the topology. A > tuple, when emitted the first time, will have an entry created in > OffsetManager. This entry will be removed only after the tuple is > successfully acknowledged and its offset successfully committed. Till then, > commits for offsets beyond the failing offset for that TopicPartition will be > blocked. > It is possible that when the spout seeks the consumer to the failing offset, > the corresponding kafka message is not returned in the poll response. This > can happen due to that offset being deleted or compacted away. In this > scenario that partition will be blocked from committing and progressing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2546) Kafka spout can stall / get stuck due to edge case with failing tuples
[ https://issues.apache.org/jira/browse/STORM-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2546: -- Fix Version/s: 2.0.0 > Kafka spout can stall / get stuck due to edge case with failing tuples > -- > > Key: STORM-2546 > URL: https://issues.apache.org/jira/browse/STORM-2546 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0 >Reporter: Prasanna Ranganathan >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The mechanism for replaying a failed tuple involves seeking the kafka > consumer to the failing offset and then re-emitting it into the topology. A > tuple, when emitted the first time, will have an entry created in > OffsetManager. This entry will be removed only after the tuple is > successfully acknowledged and its offset successfully committed. Till then, > commits for offsets beyond the failing offset for that TopicPartition will be > blocked. > It is possible that when the spout seeks the consumer to the failing offset, > the corresponding kafka message is not returned in the poll response. This > can happen due to that offset being deleted or compacted away. In this > scenario that partition will be blocked from committing and progressing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
