On 04/10/2016 03:30 AM, Erik Hugne wrote: > On Sat, Apr 09, 2016 at 02:38:15PM -0400, Jon Maloy wrote: >> TIPC based clusters are by default set up with full-mesh link >> connectivity between all nodes. Those links are expected to provide >> a short failure detection time, by default set to 1500 ms. Because >> of this, the background load for neighbor monitoring in an N-node >> cluster increases with a factor N on each node, while the overall >> monitoring traffic through the network infrastructure inceases at >> a ~(N * (N - 1)) rate. Experience has shown that such clusters don't >> scale well beyond ~100 nodes unless we significantly increase failure >> discovery tolerance. >> >> This commit introduces a framework and an algorithm that drastically >> reduces this background load, while basically maintaining the original >> failure detection times across the whole cluster. Using this algortithm, >> background load will now grow at a rate of ~(2 * sqrt(N)) per node, and >> at ~(2 * N * sqrt(N)) in traffic overhead. As an example, each node will >> now have to actively monitor 38 neighbors in a 400-node cluster, instead >> of as before 399. >> >> This "Overlapping Ring Supervision Algorithm" is completely distributed >> and employs no centralized state. It goes as follows: >> >> - Each node makes up a linearly ascending, circular list of all its >> N known neighbors, based on their TIPC node identity. This algorithm >> must be the same on all nodes. >> >> - The node then selects the next M = sqrt(N)-1 nodes downstream in the >> list, and chooses to actively monitor those. This is called its >> "local monitoring domain". > So 1.1.1 will actively monitor 1.1.2, 1.1.3,...,1.1.N > and 1.1.2 monitors 1.1.3, 1.1.4,...1.1.N+1 etc?
If you replace "N" with "M" according to the above, yes. > >> - It creates a domain record describing the monitoring domain, and >> piggy-backs this in the data area of all neigbor monitoring messages >> (LINK_PROTOCOL/STATE) leaving that node. This means that all nodes in >> the cluster eventually (default within 400 ms) will learn about >> its monitoring domain. >> >> - Whenever a node discovers a change in its local domain, e.g., a node >> has been added or has gone down, it creates and sends out a new >> version of its node record to inform all neighbors about the change. >> >> - A node receiving a domain record from anybody outside its local domain >> matches this against its own list (which may not look the same), and >> chooses to not actively monitor those members of the received domain >> record that are also present in its own list. Instead, it relies on >> indications from the direct monitoring nodes if an indirecly monitored >> node has gone up or down. If a node is indicated lost, the receiving >> node temporarily activates its own direct monitoring towards that node >> in order to confirm, or not, that it is actually gone. >> - Since each node is actively montoring sqrt(N) downstream neighbors, >> each node is also actively monitored by the same number of upstream >> neighbors. This means that all non-direct monitoring nodes normally >> will receive sqrt(N) indications that a node is gone. >> >> - A major drawback with ring monitoring is how to handle failures that >> causes massive network partitionings. If both a lost node and all its >> direct monitoring neigbors are inside the lost partition, the nodes in >> the remaining partition will never receive indications about the loss. >> To overcome this, each node also chooses to actively monitor some >> nodes outside its local domain. Those nodes are called remote domain >> "heads", and are selected in such a way that no node in the cluster >> is more than one indirect monitoring hop away. Because of this, each >> node, apart from monitoring the member of its local domain, will also >> typically monitor sqrt(N) remote head nodes. > What if each node includes it's local domain record in the DSC_REQ messages > aswell? These would be sent at 60 second intervals, meaning that a network > partitioning such as the one you described would be detected after 1min > worst case, but maybe that's too slow? Way too slow. We want to maintain the current failure discovery times if possible. We don't quite get there with this proposal; worst-case discovery time is 2 x link tolerance, i.e. 3 s. But anybody can lower the link tolerance in their system, and with this solution the risks with doing so should be much lower than they are now. > >> - As an optimization, local list status, domain status and domain >> records are marked with a generation number. This saves senders from >> unecessarily conveying unchanged domain records, and receivers from >> performing unneeded re-adaptations of their node monitoring list, such >> as re-assigning domain heads. >> >> Signed-off-by: Jon Maloy <jon.ma...@ericsson.com> > I like this, but i think it would be good if it's made configurable with > either a sysctl or bearer parameter I was thinking the same. This change is extremely intrusive for the whole cluster, and it would feel safer if we could at least disable it if somebody suspects it is the root of a problem, and the cluster is anyway of a reasonably small size. Another option is that we leave it disabled by default until the cluster reaches a pre-defined (configurable) size. Let's say that we set the default size to 30 nodes, and it can be disabled by setting it to some unrealistic big number. It might also be a good idea to be able to dump the contents of the monitor table, so we can have some insight into its contents, and that it is calculated correctly in case of doubt. Those would be bearer options, I think. Maybe something for Richard? ///jon > > //Erik > > ------------------------------------------------------------------------------ > Find and fix application performance issues faster with Applications Manager > Applications Manager provides deep performance insights into multiple tiers of > your business applications. It resolves application problems quickly and > reduces your MTTR. Get your free trial! http://pubads.g.doubleclick.net/ > gampad/clk?id=1444514301&iu=/ca-pub-7940484522588532 > _______________________________________________ > tipc-discussion mailing list > tipc-discussion@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/tipc-discussion ------------------------------------------------------------------------------ Find and fix application performance issues faster with Applications Manager Applications Manager provides deep performance insights into multiple tiers of your business applications. It resolves application problems quickly and reduces your MTTR. Get your free trial! http://pubads.g.doubleclick.net/ gampad/clk?id=1444514301&iu=/ca-pub-7940484522588532 _______________________________________________ tipc-discussion mailing list tipc-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tipc-discussion
