[ https://issues.apache.org/jira/browse/S4-110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthieu Morel updated S4-110: ------------------------------ Fix Version/s: (was: 0.6) 0.7 > Enhance cluster management features in S4 > ----------------------------------------- > > Key: S4-110 > URL: https://issues.apache.org/jira/browse/S4-110 > Project: Apache S4 > Issue Type: New Feature > Reporter: kishore gopalakrishna > Assignee: kishore gopalakrishna > Fix For: 0.7 > > > In S4 the number of partition is fixed for all streams and is dependent on > the number of nodes in the cluster. Adding new nodes to S4 cluster causes > the number of partitions to change. This results in lot of data movement. For > example if there are 4 nodes and you add another node then nearly all keys > will be remapped which result is huge data movement where as ideally only 20% > of the data should move. > By using Helix, every stream can be partitioned differently and independent > of the number of nodes. Helix distributes the partitions evenly among the > nodes. When new nodes are added, partitions can be migrated to new nodes > without changing the number of partitions and minimizes the data movement. > > In S4 handles failures by having stand by nodes that are idle most of the > time and become active when a node fails. Even though this works, its not > ideal in terms of efficient hardware usage since the stand by nodes are idle > most of the time. This also increases the fail over time since the PE state > has to be transfered to only one node. > Helix allows S4 to have Active and Standby nodes at a partition level so that > all nodes can be active but some partitions will be Active and some in stand > by mode. When a node fails, the partitions that were Active on that node > will be evenly distributed among the remaining nodes. This provides automatic > load balancing and also improves fail over time, since PE state can be > transfered to multiple nodes in parallel. > I have a prototype implementation here > https://github.com/kishoreg/incubator-s4 > Instructions to build it and try it out are in the Readme. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira