Hi Peng, Peng He <xnhp0...@gmail.com> writes:
> To seperate into N cmaps, why not use hash value divided by N? > FWIW, I think it makes sense to discuss the potential benefits of other approaches as well. They may even end up not being as performant as this one, but also some points to consider here are: - the number of zones used in the common case (or even for the specific use case as the expectation is that the fewer the zones involved, the smaller the benefit) - given flush per zone is where most of the gain is, the flush per zone for the use case As a last remark, partitioning per zone also implies a substantial design change that may potentially result in contrast with other approaches targeting the overall performance (e.g., [0] is a quick example that comes to mind with good scalability improvements in cps, and probably, but this is just a guess, measurable improvements in the same ct execute test). [0] https://patchwork.ozlabs.org/project/openvswitch/patch/165668250987.1967719.7371616138630033269.st...@fed.void/ > Simon Horman <ho...@ovn.org> 于2024年5月1日周三 19:06写道： > >> On Wed, Apr 24, 2024 at 02:44:54PM +0200, Felix Huettner via dev wrote: >> > Currently conntrack uses a single large cmap for all connections stored. >> > This cmap contains all connections for all conntrack zones which are >> > completely separate from each other. By separating each zone to its own >> > cmap we can significantly optimize the performance when using multiple >> > zones. >> > >> > The change fixes a similar issue as [1] where slow conntrack zone flush >> > operations significantly slow down OVN router failover. The difference is >> > just that this fix is used whith dpdk, while [1] was when using the ovs >> > kernel module. >> > >> > As we now need to store more cmap's the memory usage of struct conntrack >> > increases by 524280 bytes. Additionally we need 65535 cmaps with 128 >> > bytes each. This leads to a total memory increase of around 10MB. >> > >> > Running "./ovstest test-conntrack benchmark 4 33554432 32 1" shows no >> > real difference in the multithreading behaviour against a single zone. >> > >> > Running the new "./ovstest test-conntrack benchmark-zones" show >> > significant speedups as shown below. The values for "ct execute" are for >> > acting on the complete zone with all its entries in total (so in the >> > first case adding 10,000 new conntrack entries). All tests are run 1000 >> > times. >> > >> > When running with 1,000 zones with 10,000 entries each we see the >> > following results (all in microseconds): >> > "./ovstest test-conntrack benchmark-zones 10000 1000 1000" >> > >> > +------+--------+---------+---------+ >> > | Min | Max | 95%ile | Avg | >> > +------------------------+------+--------+---------+---------+ >> > | ct execute (commit) | | | | | >> > | with commit | 2266 | 3505 | 2707.06 | 2592.06 | >> > | without commit | 2411 | 12730 | 4432.50 | 2736.78 | >> > +------------------------+------+--------+---------+---------+ >> > | ct execute (no commit) | | | | | >> > | with commit | 699 | 1238 | 886.15 | 722.67 | >> > | without commit | 700 | 3377 | 1934.42 | 803.53 | >> > +------------------------+------+--------+---------+---------+ >> > | flush full zone | | | | | >> > | with commit | 619 | 1122 | 901.36 | 679.15 | >> > | without commit | 618 | 105078 | 64591 | 2886.46 | >> > +------------------------+------+--------+---------+---------+ >> > | flush empty zone | | | | | >> > | with commit | 0 | 5 | 1.00 | 0.64 | >> > | without commit | 54 | 87469 | 64520 | 2172.25 | >> > +------------------------+------+--------+---------+---------+ >> > >> > When running with 10,000 zones with 1,000 entries each we see the >> > following results (all in microseconds): >> > "./ovstest test-conntrack benchmark-zones 1000 10000 1000" >> > >> > +------+--------+---------+---------+ >> > | Min | Max | 95%ile | Avg | >> > +------------------------+------+--------+---------+---------+ >> > | ct execute (commit) | | | | | >> > | with commit | 215 | 287 | 231.88 | 222.30 | >> > | without commit | 214 | 1692 | 569.18 | 285.83 | >> > +------------------------+------+--------+---------+---------+ >> > | ct execute (no commit) | | | | | >> > | with commit | 68 | 97 | 74.69 | 70.09 | >> > | without commit | 68 | 300 | 158.40 | 82.06 | >> > +------------------------+------+--------+---------+---------+ >> > | flush full zone | | | | | >> > | with commit | 47 | 211 | 56.34 | 50.34 | >> > | without commit | 48 | 96330 | 63392 | 63923 | >> > +------------------------+------+--------+---------+---------+ >> > | flush empty zone | | | | | >> > | with commit | 0 | 1 | 1.00 | 0.44 | >> > | without commit | 3 | 109728 | 63923 | 3629.44 | >> > +------------------------+------+--------+---------+---------+ >> > >> > Comparing the averages we see: >> > * a moderate performance improvement for conntrack_execute with or >> > without commiting of around 6% to 23% >> > * a significant performance improvement for flushing a full zone of >> > around 75% to 99% >> > * an even more significant improvement for flushing empty zones since we >> > no longer need to check any unrelated connections >> >> Very nice numbers indeed. >> >> > [1] 9ec849e8aa869b646c372fac552ae2609a4b5f66 >> > >> > Signed-off-by: Felix Huettner <felix.huettner@mail.schwarz> >> >> Acked-by: Simon Horman <ho...@ovn.org> >> >> ... >> >> _______________________________________________ >> dev mailing list >> d...@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> > > > -- > hepeng > _______________________________________________ > dev mailing list > d...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
