Hi,
As we understand, this patch would bypass the node replication, so that adding
tunnel would not cause main thread to wait for workers synchronizing the nodes.
However, in addition to that, you have to do more things to be able to add 40k
or more tunnels in a predictable time period. Here is what we did for adding 2M
gtp tunnels, for your reference. Mpls tunnel should be pretty much the same.
1. Don’t call fib_entry_child_add after adding fib entry to the tunnel
(fib_table_entry_special_add ). This will create a linked list for all child
nodes belonged to the fib entry pointed to the tunnel endpoint. As a result,
adding tunnel would become slower and slower. BTW, it is not a good fix, but it
works.
#if 0
t->sibling_index = fib_entry_child_add
(t->fib_entry_index, gtm->fib_node_type, t - gtm->tunnels);
#endif
1. The bihash for Adj_nbr. Each tunnel interface would create one bihash
which by default is 32MB, mmap and memset then. Typically you don’t need that
many adjacencies for a p2p tunnel interface. We change the code to use a common
heap for all p2p interfaces
2. As mentioned in my email, rewrite requires cache line alignment, which
mheap cannot handle very well. Mheap might be super slow when you add too many
tunnels.
3. In vl_api_clnt_process, make sleep_time always 100us. This is to avoid
main thread yielding to linux_epoll_input_inline 10ms wait time. This is not a
perfect fix either. But if don’t do this, probably each API call would probably
have to wait for 10ms until main thread has chance to polling API events.
4. Be careful with the counters. It would eat up your memory very quick.
Each counter will be expanded to number of thread multiply number of tunnels.
In other words, 1M tunnels means 1M x 8 x 8B = 64MB, if you have 8 workers. The
combined counter will take double size because it has 16 bytes. Each interface
has 9 simple and 2 combined counters. Besides, load_balance_t and adjacency_t
also have some counters. You will have at least that many objects if you have
that many interfaces. The solution is simple – to make a dedicated heap for all
counters.
5. We also did some other fixes to speed up memory allocation, f.g.,
pre-allocate a big enough pool for gtpu_tunnel_t
To honest, it is not easy. It took us quite some time to figure it out. In the
end, we manage to add 2M tunnels & 2M routes in 250s.
Hope it helps.
Regard,
Kingwel
From: [email protected] [mailto:[email protected]] On Behalf Of Neale Ranns
Sent: Wednesday, April 18, 2018 4:33 PM
To: xyxue <[email protected]>; Kingwel Xie <[email protected]>
Cc: [email protected]
Subject: Re: [vpp-dev] questions in configuring tunnel
Hi Xyxue,
Try applying the changes in this patch:
https://gerrit.fd.io/r/#/c/10216/
to MPLS tunnels. Please contribute any changes back to the community so we can
all benefit.
Regards,
Neale
From: <[email protected]<mailto:[email protected]>> on behalf of xyxue
<[email protected]<mailto:[email protected]>>
Date: Wednesday, 18 April 2018 at 09:48
To: Xie <[email protected]<mailto:[email protected]>>
Cc: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: [vpp-dev] questions in configuring tunnel
Hi,
We are testing mpls tunnel.The problems shown below appear in our configuration:
1.A configuration of one tunnel will increase two node (this would lead to a
very high consumption of memory )
2.more node number, more time to update vlib_node_runtime_update and node info
traversal;
When we configured 40 thousand mpls tunnels , the configure time is 10+ minutes
, and the occurrence of out of memory.
How can you configure 2M gtpu tunnels , Can I know the configuration speed and
the memory usage?
Thanks,
Xyxue
________________________________
