[jira] [Updated] (STORM-2639) Kafka Spout incorrectly computes numCommittedOffsets due to voids in the topic (topic compaction)
[ https://issues.apache.org/jira/browse/STORM-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2639: Fix Version/s: 1.2.0 1.1.1 > Kafka Spout incorrectly computes numCommittedOffsets due to voids in the > topic (topic compaction) > - > > Key: STORM-2639 > URL: https://issues.apache.org/jira/browse/STORM-2639 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 1.1.0 >Reporter: Prasanna Ranganathan >Assignee: Prasanna Ranganathan > Fix For: 2.0.0, 1.1.1, 1.2.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > This is a followup to STORM-2505 to fix a minor issue with the computation of > numUncommittedOffsets in OffsetManager. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2541) Manual partition assignment doesn't work
[ https://issues.apache.org/jira/browse/STORM-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2541: -- Affects Version/s: 1.1.1 > Manual partition assignment doesn't work > > > Key: STORM-2541 > URL: https://issues.apache.org/jira/browse/STORM-2541 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0, 1.1.1 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Fix For: 2.0.0 > > Time Spent: 4h > Remaining Estimate: 0h > > The manual partition assignment logic in > ManualPartitionNamed/PatternSubscription is broken. The spout is unable to > start. The subscription needs to call onPartitionsAssigned even if the > current assignment is null, otherwise it becomes impossible to initialize the > spout. The order of KafkaConsumer.assign and the calls to > onPartitionsAssigned/Revoked is also wrong. The assignment must happen first, > otherwise onPartitionsAssigned will get an IllegalStateException when it > tries to call KafkaConsumer.seek on a partition the consumer is not yet > assigned. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2541) Manual partition assignment doesn't work
[ https://issues.apache.org/jira/browse/STORM-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing resolved STORM-2541. --- Resolution: Fixed > Manual partition assignment doesn't work > > > Key: STORM-2541 > URL: https://issues.apache.org/jira/browse/STORM-2541 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Fix For: 2.0.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > The manual partition assignment logic in > ManualPartitionNamed/PatternSubscription is broken. The spout is unable to > start. The subscription needs to call onPartitionsAssigned even if the > current assignment is null, otherwise it becomes impossible to initialize the > spout. The order of KafkaConsumer.assign and the calls to > onPartitionsAssigned/Revoked is also wrong. The assignment must happen first, > otherwise onPartitionsAssigned will get an IllegalStateException when it > tries to call KafkaConsumer.seek on a partition the consumer is not yet > assigned. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2133) Page-rendered-at timestamp visible on the UI
[ https://issues.apache.org/jira/browse/STORM-2133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091872#comment-16091872 ] Ethan Li commented on STORM-2133: - PR is filed at [https://github.com/apache/storm/pull/] > Page-rendered-at timestamp visible on the UI > > > Key: STORM-2133 > URL: https://issues.apache.org/jira/browse/STORM-2133 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Derek Dagit >Assignee: Ethan Li >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > As a user, I would like a simple timestamp of when a UI page was rendered, so > that, I can quickly check if the information is fresh. > This is mainly for the case of keeping browser tabs open for a long while > during debugging. Seeing old data in your browser could harm the effort. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2548) Simplify KafkaSpoutConfig
[ https://issues.apache.org/jira/browse/STORM-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2548: -- Fix Version/s: 1.2.0 2.0.0 > Simplify KafkaSpoutConfig > - > > Key: STORM-2548 > URL: https://issues.apache.org/jira/browse/STORM-2548 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Fix For: 2.0.0, 1.2.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Some suggestions for simplifying KafkaSpoutConfig off the mailing list: > * We should not duplicate properties that users would normally set in the > KafkaConsumer properties map. We should just have a setter (setProp) for > setting properties in that map. For instance, setGroupId is just duplicating > a setting that the user should be able to set directly in the consumer > properties. > * We should get rid of the key/value deserializer setters. Setting the > deserializers as classes is something the user can just as well do by using > setProp. The SerializableDeserializer class should be removed. It is only > offering extra type safety in the case where the user is defining their own > deserializer type, and has the opportunity to subclass > SerializableDeserializer. The setters don't work with the built in Kafka > deserializers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2548) Simplify KafkaSpoutConfig
[ https://issues.apache.org/jira/browse/STORM-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing resolved STORM-2548. --- Resolution: Fixed > Simplify KafkaSpoutConfig > - > > Key: STORM-2548 > URL: https://issues.apache.org/jira/browse/STORM-2548 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Fix For: 2.0.0, 1.2.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Some suggestions for simplifying KafkaSpoutConfig off the mailing list: > * We should not duplicate properties that users would normally set in the > KafkaConsumer properties map. We should just have a setter (setProp) for > setting properties in that map. For instance, setGroupId is just duplicating > a setting that the user should be able to set directly in the consumer > properties. > * We should get rid of the key/value deserializer setters. Setting the > deserializers as classes is something the user can just as well do by using > setProp. The SerializableDeserializer class should be removed. It is only > offering extra type safety in the case where the user is defining their own > deserializer type, and has the opportunity to subclass > SerializableDeserializer. The setters don't work with the built in Kafka > deserializers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2548) Simplify KafkaSpoutConfig
[ https://issues.apache.org/jira/browse/STORM-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2548: -- Affects Version/s: 1.1.0 > Simplify KafkaSpoutConfig > - > > Key: STORM-2548 > URL: https://issues.apache.org/jira/browse/STORM-2548 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Time Spent: 5h 10m > Remaining Estimate: 0h > > Some suggestions for simplifying KafkaSpoutConfig off the mailing list: > * We should not duplicate properties that users would normally set in the > KafkaConsumer properties map. We should just have a setter (setProp) for > setting properties in that map. For instance, setGroupId is just duplicating > a setting that the user should be able to set directly in the consumer > properties. > * We should get rid of the key/value deserializer setters. Setting the > deserializers as classes is something the user can just as well do by using > setProp. The SerializableDeserializer class should be removed. It is only > offering extra type safety in the case where the user is defining their own > deserializer type, and has the opportunity to subclass > SerializableDeserializer. The setters don't work with the built in Kafka > deserializers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2549) The fix for STORM-2343 is incomplete, and the spout can still get stuck on failed tuples
[ https://issues.apache.org/jira/browse/STORM-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091618#comment-16091618 ] Hugo Louro commented on STORM-2549: --- [~Srdo] can you please marked this ticket as in progress. Thanks. > The fix for STORM-2343 is incomplete, and the spout can still get stuck on > failed tuples > > > Key: STORM-2549 > URL: https://issues.apache.org/jira/browse/STORM-2549 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 2.0.0, 1.1.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Time Spent: 1.5h > Remaining Estimate: 0h > > Example: > Say maxUncommittedOffsets is 10, maxPollRecords is 5, and the committedOffset > is 0. > The spout will initially emit up to offset 10, because it is allowed to poll > until numNonRetriableTuples is >= maxUncommittedOffsets > The spout will be allowed to emit another 5 tuples if offset 10 fails, so if > that happens, offsets 10-14 will get emitted. If offset 1 fails and 2-14 get > acked, the spout gets stuck because it will count the "extra tuples" 11-14 in > numNonRetriableTuples. > An similar case is the one where maxPollRecords doesn't divide > maxUncommittedOffsets evenly. If it were 3 in the example above, the spout > might just immediately emit offsets 1-12. If 2-12 get acked, offset 1 cannot > be reemitted. > The proposed solution is the following: > * Enforce maxUncommittedOffsets on a per partition basis (i.e. actual limit > will be multiplied by the number of partitions) by always allowing poll for > retriable tuples that are within maxUncommittedOffsets tuples of the > committed offset. Pause any non-retriable partitions if the partition has > passed the maxUncommittedOffsets limit, and some other partition is polling > for retries while also at the maxUncommittedOffsets limit. > Example of this functionality: > MaxUncommittedOffsets is 100 > MaxPollRecords is 10 > Committed offset for partition 0 and 1 is 0. > Partition 0 has emitted 0 > Partition 1 has emitted 0...95, 97, 99, 101, 103 (some offsets compacted away) > Partition 1, message 99 is retriable > We check that message 99 is within 100 emitted tuples of offset 0 (it is the > 97th tuple after offset 0, so it is) > We do not pause partition 0 because that partition isn't at the > maxUncommittedOffsets limit. > Seek to offset 99 on partition 1 and poll > We get back offset 99, 101, 103 and potentially 7 new tuples. Say the lowest > of these is at offset 104. > The spout emits offset 99, filters out 101 and 103 because they were already > emitted, and emits the 7 new tuples. > If offset 104 (or later) become retriable, they are not retried until the > committed offset moves. This is because offset 104 is the 101st tuple emitted > after offset 0, so it isn't allowed to retry until the committed offset moves. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (STORM-2133) Page-rendered-at timestamp visible on the UI
[ https://issues.apache.org/jira/browse/STORM-2133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Li reassigned STORM-2133: --- Assignee: Ethan Li > Page-rendered-at timestamp visible on the UI > > > Key: STORM-2133 > URL: https://issues.apache.org/jira/browse/STORM-2133 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Derek Dagit >Assignee: Ethan Li >Priority: Minor > > As a user, I would like a simple timestamp of when a UI page was rendered, so > that, I can quickly check if the information is fresh. > This is mainly for the case of keeping browser tabs open for a long while > during debugging. Seeing old data in your browser could harm the effort. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2639) Kafka Spout incorrectly computes numCommittedOffsets due to voids in the topic (topic compaction)
[ https://issues.apache.org/jira/browse/STORM-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing resolved STORM-2639. --- Resolution: Fixed > Kafka Spout incorrectly computes numCommittedOffsets due to voids in the > topic (topic compaction) > - > > Key: STORM-2639 > URL: https://issues.apache.org/jira/browse/STORM-2639 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 1.1.0 >Reporter: Prasanna Ranganathan >Assignee: Prasanna Ranganathan > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > This is a followup to STORM-2505 to fix a minor issue with the computation of > numUncommittedOffsets in OffsetManager. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2639) Kafka Spout incorrectly computes numCommittedOffsets due to voids in the topic (topic compaction)
[ https://issues.apache.org/jira/browse/STORM-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2639: -- Affects Version/s: (was: 1.x) 1.1.0 > Kafka Spout incorrectly computes numCommittedOffsets due to voids in the > topic (topic compaction) > - > > Key: STORM-2639 > URL: https://issues.apache.org/jira/browse/STORM-2639 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 1.1.0 >Reporter: Prasanna Ranganathan >Assignee: Prasanna Ranganathan > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > This is a followup to STORM-2505 to fix a minor issue with the computation of > numUncommittedOffsets in OffsetManager. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2544) Bugs in the Kafka Spout retry logic when using manual commit
[ https://issues.apache.org/jira/browse/STORM-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2544: -- Affects Version/s: (was: 1.x) 1.1.0 > Bugs in the Kafka Spout retry logic when using manual commit > > > Key: STORM-2544 > URL: https://issues.apache.org/jira/browse/STORM-2544 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 1.1.0 >Reporter: Prasanna Ranganathan >Assignee: Prasanna Ranganathan > Fix For: 2.0.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Situation: Spout configured to use manual commit with a finite number of > retries. > In the above scenario if and when a tuple fails repeatedly and hits the retry > limit, it will neither be scheduled for an attempt again nor properly > accounted for in the ack() method and in OffsetManager. This will block > commits for the partition that the tuple belongs to. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2622) Add owner resource summary
[ https://issues.apache.org/jira/browse/STORM-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wang resolved STORM-2622. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli] Merged into master. > Add owner resource summary > -- > > Key: STORM-2622 > URL: https://issues.apache.org/jira/browse/STORM-2622 > Project: Apache Storm > Issue Type: Improvement >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > At Yahoo, we have a "Owner Summary" section in Storm UI to display the > resources a user is using and guaranteed. This work is done by [~abellina] . > I am porting the code back to apache storm. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
