[jira] [Commented] (STORM-2284) Storm Worker Redesign

2017-03-28 Thread Roshan Naik (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946420#comment-15946420
 ] 

Roshan Naik commented on STORM-2284:


Hmm... i think this will need some more thought into how that changes would be 
needed to the scheduler / elsewhere.

I am thinking .. each supervisor would need to know how many cores it is 
managing.. and each new worker gets assigned unused cores. At submit time, need 
to figure out where each spout and bolt instance goes... based on the groupings 
used.

Ya I/O threads over allocation is reasonable... we have two types of IO : 
outbound & inbound. Hard to say without experimenting and would depend on 
throughput being sustainedbut perhaps the outbound threads could be pinned 
to same core.. and inbound threads to another core.

If we decide to do memory mapped Queues for intra host messaging, then we have  
inter-host I/O and intra-host I/O... each with in/bound variations.

> Storm Worker Redesign
> -
>
> Key: STORM-2284
> URL: https://issues.apache.org/jira/browse/STORM-2284
> Project: Apache Storm
>  Issue Type: Umbrella
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>
> Much has been learnt from evolving the 1.x line. We can now use the benefit 
> of hindsight and apply these learnings into the future work on 2.x line. 
> The goal is to rethink the Worker to improve performance, enhance its 
> abilities and also retain compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (STORM-2284) Storm Worker Redesign

2017-03-27 Thread Ambud Sharma (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943640#comment-15943640
 ] 

Ambud Sharma commented on STORM-2284:
-

[~roshan_naik] for CPU pining is the idea to create a new scheduler, limit slot 
calculation based on cores or put a worker level constraint?

> Storm Worker Redesign
> -
>
> Key: STORM-2284
> URL: https://issues.apache.org/jira/browse/STORM-2284
> Project: Apache Storm
>  Issue Type: Umbrella
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>
> Much has been learnt from evolving the 1.x line. We can now use the benefit 
> of hindsight and apply these learnings into the future work on 2.x line. 
> The goal is to rethink the Worker to improve performance, enhance its 
> abilities and also retain compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (STORM-2284) Storm Worker Redesign

2017-01-11 Thread Roshan Naik (JIRA)

[ 
https://issues.apache.org/jira/browse/STORM-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820045#comment-15820045
 ] 

Roshan Naik commented on STORM-2284:


Will publish some design docs here. You can post your comments on the docs or 
in comments below. This is an umbrella jira so feel free to bring in your own 
design docs on additional areas. 

*Part 1 of the Design  document*
Also covers some motivation on why this work is important.
https://docs.google.com/document/d/1rmW_p5zyWMzYwPbArNM5z5XLhSVmCL2uDoMzcgp9tU4/edit?usp=sharing




> Storm Worker Redesign
> -
>
> Key: STORM-2284
> URL: https://issues.apache.org/jira/browse/STORM-2284
> Project: Apache Storm
>  Issue Type: Umbrella
>  Components: storm-core
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>
> Much has been learnt from evolving the 1.x line. We can now use the benefit 
> of hindsight and apply these learnings into the future work on 2.x line. 
> Here the goal is to rethink the Worker, with the goal of improving 
> performance, enhancing its abilities and also retaining compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)