> > > > So, the root cause of all the issues in this patch, in my understanding, > > is the fact that you need to collect statistics for all the bond hashes > > in order to be able to rebalance traffic. This forces you to have access > > to PMD local caches. > > > > The basic idea how to overcome this issue is to not have PMD local bond > > cache, but have an RCU-protected data structure instead. > > > > Memory layout scheme in current patch consists of 3 layers: > > 1. Global hash map for all bonds. One for the whole datapath. > > Protected by dp->bond_mutex. > > 2. Hash map of all the bonds. One for each PMD thread. > > Protected by pmd->bond_mutex. > > 3. Hash map of all the bonds. Local copy that could be used > > lockless, but only by the owning PMD thread. > > > > Suggested layout #1: > > Single global concurrent hash map (cmap) for all bonds. One for the > > whole > > datapath. Bond addition/deletion protected by the dp->bond_mutex. > > Reads are lockless since protected by RCU. Statistics updates must be > > fully atomic (i.e. atomic_add_relaxed()). > > > > Suggested layout #2: > > One cmap for each PMD thread (no global one). Bond addition/deletion > > protected by the pmd->bond_mutex. Reads are lockless since protected > > by RCU. Statistics updates should be atomic in terms of reads and > > writes. > > (non_atomic_ullong_add() function could be used). > > (This is similar to how we store per-PMD flow tables.) > > > > #1 will consume a lot less memory, but could scale worse in case of too > > many > > threads trying to send traffic to the same bond port. #2 might be a bit > > faster and more scalable in terms of performance, but less efficient in > > memory consumption and might be slower in terms of response to slave > updates > > since we have to update hash maps on all the threads. > > Both solutions doesn't require any reconfiguration of running PMD threads. > > > > Note that to update cmap entry, you will likely need to prepare the new > > cmap > > node and use a cmap_replace(). Statistics copy in this case might be a > > bit > > tricky because you may lost part of additions within the period while PMD > > threads are still using the old entry. To avoid that, statistics copying > > should be RCU-postponed. However, I'm not sure if we need highly accurate > > stats there. > >
Hi Ilya, I am inclined to use option 2 to have per-pmd cmap of bonds however, I see that even in this option we will need to have one cmap (or hmap) of bonds at global datapath level. This will take care of reconfigure_pmd_threads scenario. New PMDs needs to be configured with bond cmap from global map. Warm Regards, Vishal Ajmera _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
