[jira] [Updated] (STORM-2648) Kafka spout can't show acks/fails and complete latency when auto commit is enabled
[ https://issues.apache.org/jira/browse/STORM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2648: -- Description: The storm-kafka-client spout currently emits tuples with no message ids if auto commit is enabled. This causes the ack/fail/complete latency counters in Storm UI to be 0. In some cases this is desirable because the user may not care, and doesn't want the overhead of Storm tracking tuples. [~avermeerbergen] expressed a desire to be able to use auto commit without these counters being disabled, presumably to monitor topology performance. In order to support having acks/fails and complete latency when using auto commit, we need to make the spout always emit tuples with anchors. Since the spout doesn't care about acks or fails when auto commit is enabled, we should make the spout ack/fail methods return immediately if auto commit is enabled. I think we don't lose much by switching to this approach. Users that want the statistics in Storm UI can enable auto commit and leave topology ackers at some non-zero number. Users that don't care and don't want the overhead of having Storm track the tuples can set topology ackers to 0, which should make the spout behave a lot like it does now. The only use case I can think of where this won't work is if someone with multiple spouts in a topology needs one to be auto commit and one to be at-least-once, and they can't live with the overhead of tuple tracking for the auto commit spout. If this is a real use case, it can be worked around with an extra configuration parameter to switch whether tuples are emitted unanchored, but I'd rather keep it simple for now. was: The storm-kafka-client spout currently emits tuples with no message ids if auto commit is enabled. This causes the ack/fail/complete latency counters in Storm UI to be 0. In some cases this is desirable because the user may not care, and doesn't want the overhead of Storm tracking tuples. [~avermeerbergen] expressed a desire to be able to use auto commit without these counters being disabled, presumably to monitor topology performance(?) In order to support having acks/fails and complete latency when using auto commit, we need to make the spout always emit tuples with anchors. Since the spout doesn't care about acks or fails when auto commit is enabled, we should make the spout ack/fail methods return immediately if auto commit is enabled. I think we don't lose much by switching to this approach. Users that want the statistics in Storm UI can enable auto commit and leave topology ackers at some non-zero number. Users that don't care and don't want the overhead of having Storm track the tuples can set topology ackers to 0, which should make the spout behave a lot like it does now. The only use case I can think of where this won't work is if someone with multiple spouts in a topology needs one to be auto commit and one to be at-least-once, and they can't live with the overhead of tuple tracking for the auto commit spout. If this is a real use case, it can be worked around with an extra configuration parameter to switch whether tuples are emitted unanchored, but I'd rather keep it simple for now. > Kafka spout can't show acks/fails and complete latency when auto commit is > enabled > -- > > Key: STORM-2648 > URL: https://issues.apache.org/jira/browse/STORM-2648 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0, 1.2.0 >Reporter: Stig Rohde Døssing >Priority: Minor > Fix For: 1.1.1 > > > The storm-kafka-client spout currently emits tuples with no message ids if > auto commit is enabled. This causes the ack/fail/complete latency counters in > Storm UI to be 0. In some cases this is desirable because the user may not > care, and doesn't want the overhead of Storm tracking tuples. > [~avermeerbergen] expressed a desire to be able to use auto commit without > these counters being disabled, presumably to monitor topology performance. > In order to support having acks/fails and complete latency when using auto > commit, we need to make the spout always emit tuples with anchors. Since the > spout doesn't care about acks or fails when auto commit is enabled, we should > make the spout ack/fail methods return immediately if auto commit is enabled. > I think we don't lose much by switching to this approach. Users that want the > statistics in Storm UI can enable auto commit and leave topology ackers at > some non-zero number. Users that don't care and don't want the overhead of > having Storm track the tuples can set topology ackers to 0, which should make > the spout behave a lot like it does now. > The on
[jira] [Updated] (STORM-2648) Kafka spout can't show acks/fails and complete latency when auto commit is enabled
[ https://issues.apache.org/jira/browse/STORM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2648: -- Fix Version/s: (was: 1.1.1) > Kafka spout can't show acks/fails and complete latency when auto commit is > enabled > -- > > Key: STORM-2648 > URL: https://issues.apache.org/jira/browse/STORM-2648 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0, 1.2.0 >Reporter: Stig Rohde Døssing >Priority: Minor > > The storm-kafka-client spout currently emits tuples with no message ids if > auto commit is enabled. This causes the ack/fail/complete latency counters in > Storm UI to be 0. In some cases this is desirable because the user may not > care, and doesn't want the overhead of Storm tracking tuples. > [~avermeerbergen] expressed a desire to be able to use auto commit without > these counters being disabled, presumably to monitor topology performance. > In order to support having acks/fails and complete latency when using auto > commit, we need to make the spout always emit tuples with anchors. Since the > spout doesn't care about acks or fails when auto commit is enabled, we should > make the spout ack/fail methods return immediately if auto commit is enabled. > I think we don't lose much by switching to this approach. Users that want the > statistics in Storm UI can enable auto commit and leave topology ackers at > some non-zero number. Users that don't care and don't want the overhead of > having Storm track the tuples can set topology ackers to 0, which should make > the spout behave a lot like it does now. > The only use case I can think of where this won't work is if someone with > multiple spouts in a topology needs one to be auto commit and one to be > at-least-once, and they can't live with the overhead of tuple tracking for > the auto commit spout. If this is a real use case, it can be worked around > with an extra configuration parameter to switch whether tuples are emitted > unanchored, but I'd rather keep it simple for now. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2648) Kafka spout can't show acks/fails and complete latency when auto commit is enabled
[ https://issues.apache.org/jira/browse/STORM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2648: -- Description: The storm-kafka-client spout currently emits tuples with no message ids if auto commit is enabled. This causes the ack/fail/complete latency counters in Storm UI to be 0. In some cases this is desirable because the user may not care, and doesn't want the overhead of Storm tracking tuples. [~avermeerbergen] expressed a desire to be able to use auto commit without these counters being disabled, presumably to monitor topology performance. We should add a toggle that allows users to enable/disable tuple anchoring in the auto commit case. was: The storm-kafka-client spout currently emits tuples with no message ids if auto commit is enabled. This causes the ack/fail/complete latency counters in Storm UI to be 0. In some cases this is desirable because the user may not care, and doesn't want the overhead of Storm tracking tuples. [~avermeerbergen] expressed a desire to be able to use auto commit without these counters being disabled, presumably to monitor topology performance. In order to support having acks/fails and complete latency when using auto commit, we need to make the spout always emit tuples with anchors. Since the spout doesn't care about acks or fails when auto commit is enabled, we should make the spout ack/fail methods return immediately if auto commit is enabled. I think we don't lose much by switching to this approach. Users that want the statistics in Storm UI can enable auto commit and leave topology ackers at some non-zero number. Users that don't care and don't want the overhead of having Storm track the tuples can set topology ackers to 0, which should make the spout behave a lot like it does now. The only use case I can think of where this won't work is if someone with multiple spouts in a topology needs one to be auto commit and one to be at-least-once, and they can't live with the overhead of tuple tracking for the auto commit spout. If this is a real use case, it can be worked around with an extra configuration parameter to switch whether tuples are emitted unanchored, but I'd rather keep it simple for now. > Kafka spout can't show acks/fails and complete latency when auto commit is > enabled > -- > > Key: STORM-2648 > URL: https://issues.apache.org/jira/browse/STORM-2648 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > The storm-kafka-client spout currently emits tuples with no message ids if > auto commit is enabled. This causes the ack/fail/complete latency counters in > Storm UI to be 0. In some cases this is desirable because the user may not > care, and doesn't want the overhead of Storm tracking tuples. > [~avermeerbergen] expressed a desire to be able to use auto commit without > these counters being disabled, presumably to monitor topology performance. > We should add a toggle that allows users to enable/disable tuple anchoring in > the auto commit case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2648) Kafka spout can't show acks/fails and complete latency when auto commit is enabled
[ https://issues.apache.org/jira/browse/STORM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated STORM-2648: -- Labels: pull-request-available (was: ) > Kafka spout can't show acks/fails and complete latency when auto commit is > enabled > -- > > Key: STORM-2648 > URL: https://issues.apache.org/jira/browse/STORM-2648 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > The storm-kafka-client spout currently emits tuples with no message ids if > auto commit is enabled. This causes the ack/fail/complete latency counters in > Storm UI to be 0. In some cases this is desirable because the user may not > care, and doesn't want the overhead of Storm tracking tuples. > [~avermeerbergen] expressed a desire to be able to use auto commit without > these counters being disabled, presumably to monitor topology performance. > We should add a toggle that allows users to enable/disable tuple anchoring in > the auto commit case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2648) Kafka spout can't show acks/fails and complete latency when auto commit is enabled
[ https://issues.apache.org/jira/browse/STORM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2648: Fix Version/s: 2.0.0 > Kafka spout can't show acks/fails and complete latency when auto commit is > enabled > -- > > Key: STORM-2648 > URL: https://issues.apache.org/jira/browse/STORM-2648 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > The storm-kafka-client spout currently emits tuples with no message ids if > auto commit is enabled. This causes the ack/fail/complete latency counters in > Storm UI to be 0. In some cases this is desirable because the user may not > care, and doesn't want the overhead of Storm tracking tuples. > [~avermeerbergen] expressed a desire to be able to use auto commit without > these counters being disabled, presumably to monitor topology performance. > We should add a toggle that allows users to enable/disable tuple anchoring in > the auto commit case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
