[jira] [Commented] (STORM-2284) Storm Worker Redesign
[ https://issues.apache.org/jira/browse/STORM-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946420#comment-15946420 ] Roshan Naik commented on STORM-2284: Hmm... i think this will need some more thought into how that changes would be needed to the scheduler / elsewhere. I am thinking .. each supervisor would need to know how many cores it is managing.. and each new worker gets assigned unused cores. At submit time, need to figure out where each spout and bolt instance goes... based on the groupings used. Ya I/O threads over allocation is reasonable... we have two types of IO : outbound & inbound. Hard to say without experimenting and would depend on throughput being sustainedbut perhaps the outbound threads could be pinned to same core.. and inbound threads to another core. If we decide to do memory mapped Queues for intra host messaging, then we have inter-host I/O and intra-host I/O... each with in/bound variations. > Storm Worker Redesign > - > > Key: STORM-2284 > URL: https://issues.apache.org/jira/browse/STORM-2284 > Project: Apache Storm > Issue Type: Umbrella > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > > Much has been learnt from evolving the 1.x line. We can now use the benefit > of hindsight and apply these learnings into the future work on 2.x line. > The goal is to rethink the Worker to improve performance, enhance its > abilities and also retain compatibility. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (STORM-2284) Storm Worker Redesign
[ https://issues.apache.org/jira/browse/STORM-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943640#comment-15943640 ] Ambud Sharma commented on STORM-2284: - [~roshan_naik] for CPU pining is the idea to create a new scheduler, limit slot calculation based on cores or put a worker level constraint? > Storm Worker Redesign > - > > Key: STORM-2284 > URL: https://issues.apache.org/jira/browse/STORM-2284 > Project: Apache Storm > Issue Type: Umbrella > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > > Much has been learnt from evolving the 1.x line. We can now use the benefit > of hindsight and apply these learnings into the future work on 2.x line. > The goal is to rethink the Worker to improve performance, enhance its > abilities and also retain compatibility. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (STORM-2284) Storm Worker Redesign
[ https://issues.apache.org/jira/browse/STORM-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820045#comment-15820045 ] Roshan Naik commented on STORM-2284: Will publish some design docs here. You can post your comments on the docs or in comments below. This is an umbrella jira so feel free to bring in your own design docs on additional areas. *Part 1 of the Design document* Also covers some motivation on why this work is important. https://docs.google.com/document/d/1rmW_p5zyWMzYwPbArNM5z5XLhSVmCL2uDoMzcgp9tU4/edit?usp=sharing > Storm Worker Redesign > - > > Key: STORM-2284 > URL: https://issues.apache.org/jira/browse/STORM-2284 > Project: Apache Storm > Issue Type: Umbrella > Components: storm-core >Reporter: Roshan Naik >Assignee: Roshan Naik > > Much has been learnt from evolving the 1.x line. We can now use the benefit > of hindsight and apply these learnings into the future work on 2.x line. > Here the goal is to rethink the Worker, with the goal of improving > performance, enhancing its abilities and also retaining compatibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)