When a cluster is started, there is often no ordering or rate limitation of how and when the individual nodes are starting. Since the initial setup between nodes is accompanied by some message exchange, this may lead to significant traffic bursts when many nodes are nodes are interconnecting simultaneously. This has turned out to be a problem in very large clusters, such as a multi-namespace environments with hundreds of virtual nodes.
In this commit, we introduce a rate control for when a node can accept new neighbors, based on the knowledge that an ignored neighbor discovery message will be followed by others within an acceptable time frame. The rate control algorithm is very modest. When the 50'th neighbor is added to a node, the latter won't accept any new neigbors until an "acceptance interval" of 20 microseconds has passed. Thereafter it adds another 20 micoseconds to this interval for each new neighbor This means that e.g., the 500'th node will not be accepted until 10 ms after the previous one was. This seems to be sufficient to solve the problem with cluster startup overload, while adding only a few seconds to the overall cluster establishment time. Signed-off-by: Jon Maloy <[email protected]> --- net/tipc/core.h | 1 + net/tipc/discover.c | 19 ++++++++++++++++++- net/tipc/node.c | 6 +++++- 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/net/tipc/core.h b/net/tipc/core.h index a1845fb..2e359ab 100644 --- a/net/tipc/core.h +++ b/net/tipc/core.h @@ -89,6 +89,7 @@ struct tipc_net { struct list_head node_list; u32 num_nodes; u32 num_links; + unsigned long prev_setup; /* Neighbor monitoring list */ struct tipc_monitor *monitors[MAX_BEARERS]; diff --git a/net/tipc/discover.c b/net/tipc/discover.c index ad9d477..391dc85 100644 --- a/net/tipc/discover.c +++ b/net/tipc/discover.c @@ -114,6 +114,22 @@ static void disc_dupl_alert(struct tipc_bearer *b, u32 node_addr, media_addr_str, b->name); } +/* disc_setup_permitted - rate control addition of neighbor nodes + */ +static bool disc_setup_permitted(struct net *net) +{ + struct tipc_net *tn = tipc_net(net); + unsigned long accept_intv; + unsigned long now = jiffies_to_msecs(jiffies); + + /* Increase accept interval with 1 ms for each 50th added node */ + accept_intv = tn->num_nodes / 50; + if ((now - tn->prev_setup) < accept_intv) + return false; + tn->prev_setup = now; + return true; +} + /** * tipc_disc_rcv - handle incoming discovery message (request or response) * @net: the applicable net namespace @@ -158,7 +174,8 @@ void tipc_disc_rcv(struct net *net, struct sk_buff *skb, return; if (!tipc_in_scope(bearer->domain, onode)) return; - + if ((mtyp == DSC_REQ_MSG) && !disc_setup_permitted(net)) + return; tipc_node_check_dest(net, onode, bearer, caps, signature, &maddr, &respond, &dupl_addr); if (dupl_addr) diff --git a/net/tipc/node.c b/net/tipc/node.c index e852c55..79670e7 100644 --- a/net/tipc/node.c +++ b/net/tipc/node.c @@ -323,6 +323,8 @@ struct tipc_node *tipc_node_create(struct net *net, u32 addr, u16 capabilities) pr_warn("Node creation failed, no memory\n"); goto exit; } + tn->num_nodes++; + tn->prev_setup = jiffies_to_msecs(jiffies); n->addr = addr; n->net = net; n->capabilities = capabilities; @@ -384,10 +386,12 @@ static void tipc_node_calculate_timer(struct tipc_node *n, struct tipc_link *l) static void tipc_node_delete(struct tipc_node *node) { + struct tipc_net *tn = tipc_net(node->net); + list_del_rcu(&node->list); hlist_del_rcu(&node->hash); tipc_node_put(node); - + tn->num_nodes--; del_timer_sync(&node->timer); tipc_node_put(node); } -- 1.9.1 ------------------------------------------------------------------------------ Mobile security can be enabling, not merely restricting. Employees who bring their own devices (BYOD) to work are irked by the imposition of MDM restrictions. Mobile Device Manager Plus allows you to control only the apps on BYO-devices by containerizing them, leaving personal data untouched! https://ad.doubleclick.net/ddm/clk/304595813;131938128;j _______________________________________________ tipc-discussion mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tipc-discussion
