[ https://issues.apache.org/jira/browse/STORM-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans resolved STORM-2539. ---------------------------------------- Resolution: Duplicate [~siwoon.son], There is a pull request up for something like this on STORM-2686, so I am marking this as a duplicate. > Locality aware grouping, which is a new grouping method considering locality > ---------------------------------------------------------------------------- > > Key: STORM-2539 > URL: https://issues.apache.org/jira/browse/STORM-2539 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core > Reporter: Siwoon Son > Attachments: receiver-imbalance.png, sender-imbalance.png > > > I’d like to propose a new grouping method. This method solves problems that > can occur with existing _Shuffle grouping_ method and _Local-or-shuffle_ > method. > I was motivated by the following a question. > bq. "Would not it be more efficient to transfer tuples to nearby nodes?" > Among the Storm's grouping methods, _Shuffle grouping_ is a method of > randomly selecting a task of the next node. In this method, all nodes can > receive data evenly, but the same amount of data is transferred to a > relatively distant node, which can cause a high delay. To solve this problem, > Storm provides Local-or-shuffle grouping considering locality. > _Local-or-shuffle grouping_ can minimize the delay by internally processing > data in the process if the task receiving the data is in the same process. > Local-or-shuffle grouping, however, considers *only locality*, which may > leads to two load balancing problems: +the sender imbalance+ and +the > receiver imbalance+. First, the sender imbalance problem is a load balancing > problem in which traffic is concentrated on a specific task because the > number of senders’ tasks and the number of nodes are not equal. Next, the > receiver imbalance problem is a load balancing problem in which traffic is > concentrated on a specific object because the number of receivers' tasks and > the number of nodes are not equal. If these problems occur, the tasks of > receivers will perform different amounts of work, resulting in performance > degradation and processing delays. > |!sender-imbalance.png|width=400!|!receiver-imbalance.png|width=400!| > |(a) Example of sender imbalance problem.|(b) Example of receiver imbalance > problem.| > So, I propose locality aware grouping which can solve the load balancing > problem while considering the locality. Locality aware grouping is a method > of periodically calculating the ping response time between nodes and > transmitting more tuples probabilistically to nodes with low response time. I > implemented the proposed Locality aware grouping at > [https://github.com/dke-knu/i2am/tree/master/i2am-app/locality-aware-grouping]. > LocalityAwareGrouping.java is a class that implements locality aware > grouping. LocalityGroupingTestTopology.java, TupleGeneratorSpout.java, and > PerformanceLoggingBolt.java are topology classes for testing this. > LocalityAwareGrouping$prepare() method reads the network information of each > node from the Zookeeper and activates the thread. This thread periodically > calculates the ping response time of each node. > LocalityAwareGrouping$chooseTasks() method selects a task by a higher > probability for the nodes with lower network response times. > But the implementation is straightforward. To calculate the ping between > nodes, the network information of the nodes that tasks are performing is > needed. I got this information from Zookeeper using the Zookeeper$getData() > method. At this time, in order to create a Zookeeper object, I had no choice > but to receive the connection information of the Zookeeper from the user. > Please let me know, if there is a way to get the network information of each > node without requiring additional parameters from the user and if you have > any additional comments. > Thank you for reading. -- This message was sent by Atlassian JIRA (v6.4.14#64029)