After quickly adding some more prints into the testsuite.
Test 1: Without 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=yes --- Maximum (NB in msec): 1130 Average (NB in msec): 620.375000 Maximum (SB in msec): 23 Average (SB in msec): 21.468759 Maximum (northd-loop in msec): 6002 Minimum (northd-loop in msec): 0 Average (northd-loop in msec): 914.760417 Long term average (northd-loop in msec): 104.799340 With 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=yes --- Maximum (NB in msec): 1148 Average (NB in msec): 630.250000 Maximum (SB in msec): 24 Average (SB in msec): 21.468744 Maximum (northd-loop in msec): 6090 Minimum (northd-loop in msec): 0 Average (northd-loop in msec): 762.101565 Long term average (northd-loop in msec): 80.735192 The metric which actually matters and which SHOULD me measured - long term average is better by 20%. Using short term average instead of long term in the test suite is actually a BUG. Are you running yours under some sort of virtualization? A. On 30/09/2021 07:52, Han Zhou wrote:
Thanks Anton for checking. I am using: Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz, 24 cores. It is weird why my result is so different. I also verified with a scale test script that creates a large scale NB/SB with 800 nodes of simulated k8s setup. And then just run: ovn-nbctl --print-wait-time --wait=sb sync Without parallel: ovn-northd completion: 7807ms With parallel: ovn-northd completion: 41267ms I suspected the hmap size problem but I tried changing the initial size to 64k buckets and it didn't help. I will find some time to check the "perf" reports. Thanks, Han On Wed, Sep 29, 2021 at 11:31 PM Anton Ivanov <anton.iva...@cambridgegreys.com <mailto:anton.iva...@cambridgegreys.com>> wrote: On 30/09/2021 07:16, Anton Ivanov wrote:Results on a Ryzen 5 3600 - 6 cores 12 threadsI will also have a look into the "maximum" measurement for multi-thread. It does not tie up with the drop in average across the board. A.Without 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=yes --- Maximum (NB in msec): 1256 Average (NB in msec): 679.463785 Maximum (SB in msec): 25 Average (SB in msec): 22.489798 Maximum (northd-loop in msec): 1347 Average (northd-loop in msec): 799.944878 2: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=no --- Maximum (NB in msec): 1956 Average (NB in msec): 809.387285 Maximum (SB in msec): 24 Average (SB in msec): 21.649258 Maximum (northd-loop in msec): 2011 Average (northd-loop in msec): 961.718686 5: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=yes --- Maximum (NB in msec): 557 Average (NB in msec): 474.010337 Maximum (SB in msec): 15 Average (SB in msec): 13.927192 Maximum (northd-loop in msec): 1261 Average (northd-loop in msec): 580.999122 6: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=no --- Maximum (NB in msec): 756 Average (NB in msec): 625.614724 Maximum (SB in msec): 15 Average (SB in msec): 14.181048 Maximum (northd-loop in msec): 1649 Average (northd-loop in msec): 746.208332 With 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=yes --- Maximum (NB in msec): 1140 Average (NB in msec): 631.125000 Maximum (SB in msec): 24 Average (SB in msec): 21.453609 Maximum (northd-loop in msec): 6080 Average (northd-loop in msec): 759.718815 2: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=no --- Maximum (NB in msec): 1210 Average (NB in msec): 673.000000 Maximum (SB in msec): 27 Average (SB in msec): 22.453125 Maximum (northd-loop in msec): 6514 Average (northd-loop in msec): 808.596842 5: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=yes --- Maximum (NB in msec): 798 Average (NB in msec): 429.750000 Maximum (SB in msec): 15 Average (SB in msec): 12.998533 Maximum (northd-loop in msec): 3835 Average (northd-loop in msec): 564.875986 6: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical Ports/Hypervisor -- ovn-northd -- dp-groups=no --- Maximum (NB in msec): 1074 Average (NB in msec): 593.875000 Maximum (SB in msec): 14 Average (SB in msec): 13.655273 Maximum (northd-loop in msec): 4973 Average (northd-loop in msec): 771.102605 The only one slower is test 6 which I will look into. The rest are > 5% faster. A. On 30/09/2021 00:56, Han Zhou wrote:On Wed, Sep 15, 2021 at 5:45 AM <anton.iva...@cambridgegreys.com <mailto:anton.iva...@cambridgegreys.com>> wrote: > > From: Anton Ivanov <anton.iva...@cambridgegreys.com <mailto:anton.iva...@cambridgegreys.com>> > > Restore parallel build with dp groups using rwlock instead > of per row locking as an underlying mechanism. > > This provides improvement ~ 10% end-to-end on ovn-heater > under virutalization despite awakening some qemu gremlin > which makes qemu climb to silly CPU usage. The gain on > bare metal is likely to be higher. > Hi Anton, I am trying to see the benefit of parallel_build, but encountered unexpected performance result when running the perf tests with command: make check-perf TESTSUITEFLAGS="--rebuild" It shows significantly worse performance than without parallel_build. For dp_group = no cases, it is better, but still ~30% slower than without parallel_build. I have 24 cores, but each thread is not consuming much CPU except the main thread. I also tried hardcode the number of thread to just 4, which end up with slightly better results, but still far behind "without parallel_build". no parallel | parallel (24 pool threads) | parallel with (4 pool threads) | | 1: ovn-northd basic scale test -- 200 Hypervisors, 200 | 1: ovn-northd basic scale test -- 200 Hypervisors, 200 | 1: ovn-northd basic scale test -- 200 Hypervisors, 200 --- | --- | --- Maximum (NB in msec): 1058 | Maximum (NB in msec): 4269 | Maximum (NB in msec): 4097 Average (NB in msec): 836.941167 | Average (NB in msec): 3697.253931 | Average (NB in msec): 3498.311525 Maximum (SB in msec): 30 | Maximum (SB in msec): 30 | Maximum (SB in msec): 28 Average (SB in msec): 25.934011 | Average (SB in msec): 26.001840 | Average (SB in msec): 25.685091 Maximum (northd-loop in msec): 1204 | Maximum (northd-loop in msec): 4379 | Maximum (northd-loop in msec): 4251 Average (northd-loop in msec): 1005.330078 | Average (northd-loop in msec): 4233.871504 | Average (northd-loop in msec): 4022.774208 | | 2: ovn-northd basic scale test -- 200 Hypervisors, 200 | 2: ovn-northd basic scale test -- 200 Hypervisors, 200 | 2: ovn-northd basic scale test -- 200 Hypervisors, 200 --- | --- | --- Maximum (NB in msec): 1124 | Maximum (NB in msec): 1480 | Maximum (NB in msec): 1331 Average (NB in msec): 892.403405 | Average (NB in msec): 1206.189287 | Average (NB in msec): 1089.378455 Maximum (SB in msec): 29 | Maximum (SB in msec): 31 | Maximum (SB in msec): 30 Average (SB in msec): 26.922632 | Average (SB in msec): 26.636706 | Average (SB in msec): 25.657484 Maximum (northd-loop in msec): 1275 | Maximum (northd-loop in msec): 1639 | Maximum (northd-loop in msec): 1495 Average (northd-loop in msec): 1074.917873 | Average (northd-loop in msec): 1458.152327 | Average (northd-loop in msec): 1301.057201 | | 5: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5: ovn-northd basic scale test -- 500 Hypervisors, 50 --- | --- | --- Maximum (NB in msec): 768 | Maximum (NB in msec): 3086 | Maximum (NB in msec): 2876 Average (NB in msec): 614.491938 | Average (NB in msec): 2681.688365 | Average (NB in msec): 2531.255444 Maximum (SB in msec): 18 | Maximum (SB in msec): 17 | Maximum (SB in msec): 18 Average (SB in msec): 16.347526 | Average (SB in msec): 15.955263 | Average (SB in msec): 16.278075 Maximum (northd-loop in msec): 889 | Maximum (northd-loop in msec): 3247 | Maximum (northd-loop in msec): 3031 Average (northd-loop in msec): 772.083572 | Average (northd-loop in msec): 3117.504297 | Average (northd-loop in msec): 2833.182361 | | 6: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6: ovn-northd basic scale test -- 500 Hypervisors, 50 --- | --- | --- Maximum (NB in msec): 1046 | Maximum (NB in msec): 1371 | Maximum (NB in msec): 1262 Average (NB in msec): 827.735852 | Average (NB in msec): 1135.514228 | Average (NB in msec): 970.544792 Maximum (SB in msec): 19 | Maximum (SB in msec): 18 | Maximum (SB in msec): 19 Average (SB in msec): 16.828127 | Average (SB in msec): 16.083914 | Average (SB in msec): 15.602525 Maximum (northd-loop in msec): 1163 | Maximum (northd-loop in msec): 1545 | Maximum (northd-loop in msec): 1411 Average (northd-loop in msec): 972.567407 | Average (northd-loop in msec): 1328.617583 | Average (northd-loop in msec): 1207.667100 I didn't debug yet, but do you have any clue what could be the reason? I am using the upstream commit 9242f27f63 which already included this patch. Below is my change to the perf-northd.at <http://perf-northd.at> file just to enable parallel_build: diff --git a/tests/perf-northd.at <http://perf-northd.at> b/tests/perf-northd.at <http://perf-northd.at> index 74b69e9d4..9328c2e21 100644 --- a/tests/perf-northd.at <http://perf-northd.at> +++ b/tests/perf-northd.at <http://perf-northd.at> @@ -191,6 +191,7 @@ AT_SETUP([ovn-northd basic scale test -- 200 Hypervisors, 200 Logical Ports/Hype PERF_RECORD_START() ovn_start +ovn-nbctl set nb_global . options:use_parallel_build=true BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(200, 200)) @@ -203,9 +204,10 @@ AT_SETUP([ovn-northd basic scale test -- 500 Hypervisors, 50 Logical Ports/Hyper PERF_RECORD_START() ovn_start +ovn-nbctl set nb_global . options:use_parallel_build=true BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(500, 50)) Thanks, Han-- Anton R. IvanovCambridgegreys Limited. Registered in England. Company Number 10273661 https://www.cambridgegreys.com/ <https://www.cambridgegreys.com/>-- Anton R. IvanovCambridgegreys Limited. Registered in England. Company Number 10273661 https://www.cambridgegreys.com/ <https://www.cambridgegreys.com/>
-- Anton R. Ivanov Cambridgegreys Limited. Registered in England. Company Number 10273661 https://www.cambridgegreys.com/ _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
