[jira] [Created] (STORM-3313) Batch support for JdbcInsertBolt was not merged to master branch
abc created STORM-3313: -- Summary: Batch support for JdbcInsertBolt was not merged to master branch Key: STORM-3313 URL: https://issues.apache.org/jira/browse/STORM-3313 Project: Apache Storm Issue Type: Bug Reporter: abc There was some work done for batching in JdbcInsertBolt https://issues.apache.org/jira/browse/STORM-1957?jql=project%20%3D%20STORM But wondering why it hasn't been merged to master branch. I am using storm-jdbc 1.2.2 (latest) but it does not have batching. Please fix this -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2359) Revising Message Timeouts
[ https://issues.apache.org/jira/browse/STORM-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739639#comment-16739639 ] Stig Rohde Døssing commented on STORM-2359: --- I've been taking a look at the feasibility of automatically resetting timeouts for tuples that are still being processed, and I think we can do it without much overhead. The idea is to track the anchor ids of each non-system message that enters an executor in/out queue. For the inbound queue, the anchor is no longer in progress when the associated tuple is acked or failed. For the outbound queue (pendingEmits in the Executor), the anchor is no longer in progress when the associated tuple gets flushed from pendingEmits. Occasionally a thread will check the set of in-progress anchors for the worker and send reset messages for all of them to the relevant ackers. In order to avoid sending too many messages, this thread snapshots the anchor set when it runs, and only sends reset messages for anchors that have been in progress sufficiently long in that worker. Since there may be more than one tuple per anchor, anchors are tracked as a count in a multiset, rather than just presence in a set. I've updated the spreadsheet with benchmark numbers for TVL with this functionality enabled. For the 90k example I also did a run where the grace period is disabled, to show the penalty for sending resets in the worst case, i.e. all in progress tuples have their timeouts reset every time the resetter thread runs. The code is available at https://github.com/srdo/storm/tree/auto-reset-timeout. Only the latest commit is new. > Revising Message Timeouts > - > > Key: STORM-2359 > URL: https://issues.apache.org/jira/browse/STORM-2359 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Stig Rohde Døssing >Priority: Major > Attachments: STORM-2359.ods, STORM-2359.ods > > > A revised strategy for message timeouts is proposed here. > Design Doc: > > https://docs.google.com/document/d/1am1kO7Wmf17U_Vz5_uyBB2OuSsc4TZQWRvbRhX52n5w/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2359) Revising Message Timeouts
[ https://issues.apache.org/jira/browse/STORM-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2359: -- Attachment: STORM-2359.ods > Revising Message Timeouts > - > > Key: STORM-2359 > URL: https://issues.apache.org/jira/browse/STORM-2359 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Stig Rohde Døssing >Priority: Major > Attachments: STORM-2359.ods, STORM-2359.ods > > > A revised strategy for message timeouts is proposed here. > Design Doc: > > https://docs.google.com/document/d/1am1kO7Wmf17U_Vz5_uyBB2OuSsc4TZQWRvbRhX52n5w/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2359) Revising Message Timeouts
[ https://issues.apache.org/jira/browse/STORM-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2359: -- Attachment: (was: STORM-2359.ods) > Revising Message Timeouts > - > > Key: STORM-2359 > URL: https://issues.apache.org/jira/browse/STORM-2359 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Stig Rohde Døssing >Priority: Major > Attachments: STORM-2359-with-auto-reset.ods, STORM-2359.ods > > > A revised strategy for message timeouts is proposed here. > Design Doc: > > https://docs.google.com/document/d/1am1kO7Wmf17U_Vz5_uyBB2OuSsc4TZQWRvbRhX52n5w/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2359) Revising Message Timeouts
[ https://issues.apache.org/jira/browse/STORM-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2359: -- Attachment: STORM-2359-with-auto-reset.ods > Revising Message Timeouts > - > > Key: STORM-2359 > URL: https://issues.apache.org/jira/browse/STORM-2359 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Stig Rohde Døssing >Priority: Major > Attachments: STORM-2359-with-auto-reset.ods, STORM-2359.ods > > > A revised strategy for message timeouts is proposed here. > Design Doc: > > https://docs.google.com/document/d/1am1kO7Wmf17U_Vz5_uyBB2OuSsc4TZQWRvbRhX52n5w/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2359) Revising Message Timeouts
[ https://issues.apache.org/jira/browse/STORM-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739641#comment-16739641 ] Stig Rohde Døssing commented on STORM-2359: --- I think it makes sense to open a WIP PR for further discussion? > Revising Message Timeouts > - > > Key: STORM-2359 > URL: https://issues.apache.org/jira/browse/STORM-2359 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Stig Rohde Døssing >Priority: Major > Attachments: STORM-2359-with-auto-reset.ods, STORM-2359.ods > > > A revised strategy for message timeouts is proposed here. > Design Doc: > > https://docs.google.com/document/d/1am1kO7Wmf17U_Vz5_uyBB2OuSsc4TZQWRvbRhX52n5w/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3313) Batch support for JdbcInsertBolt was not merged to master branch
[ https://issues.apache.org/jira/browse/STORM-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] abc updated STORM-3313: --- Issue Type: New Feature (was: Bug) > Batch support for JdbcInsertBolt was not merged to master branch > > > Key: STORM-3313 > URL: https://issues.apache.org/jira/browse/STORM-3313 > Project: Apache Storm > Issue Type: New Feature >Reporter: abc >Priority: Critical > > There was some work done for batching in JdbcInsertBolt > https://issues.apache.org/jira/browse/STORM-1957?jql=project%20%3D%20STORM > But wondering why it hasn't been merged to master branch. > I am using storm-jdbc 1.2.2 (latest) but it does not have batching. Please > fix this -- This message was sent by Atlassian JIRA (v7.6.3#76005)
