On 2020-10-01 09:39, Michal Vančo via freebsd-net wrote:
Hi
Hi Michal,
I have a server with one Mellanox ConnectX-4 adapter and the following CPU configuration (SMT disabled): # dmesg | grep SMP FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 hardware threads FreeBSD/SMP Online: 2 package(s) x 8 core(s) What I don't understand is the number of IRQs allocated for each mlx5_core: # vmstat -i | grep mlx5_core irq320: mlx5_core0 1 0 irq321: mlx5_core0 18646775 84 irq322: mlx5_core0 21 0 irq323: mlx5_core0 97793 0 irq324: mlx5_core0 84685 0 irq325: mlx5_core0 89288 0 irq326: mlx5_core0 93564 0 irq327: mlx5_core0 86892 0 irq328: mlx5_core0 99141 0 irq329: mlx5_core0 86695 0 irq330: mlx5_core0 104023 0 irq331: mlx5_core0 85238 0 irq332: mlx5_core0 88387 0 irq333: mlx5_core0 93310221 420
^^^ it appears you have some application which is using a single TCP connection heavily. Then the traffic doesn't get distributed.
irq334: mlx5_core0 1135906 5 irq335: mlx5_core0 85394 0 irq336: mlx5_core0 88361 0 irq337: mlx5_core0 88826 0 irq338: mlx5_core0 17909515 81 irq339: mlx5_core1 1 0 irq340: mlx5_core1 18646948 84 irq341: mlx5_core1 25 0 irq342: mlx5_core1 208684 1 irq343: mlx5_core1 91567 0 irq344: mlx5_core1 88340 0 irq345: mlx5_core1 92597 0 irq346: mlx5_core1 85108 0 irq347: mlx5_core1 98858 0 irq348: mlx5_core1 88103 0 irq349: mlx5_core1 104906 0 irq350: mlx5_core1 84947 0 irq351: mlx5_core1 99767 0 irq352: mlx5_core1 9482571 43 irq353: mlx5_core1 1724267 8 irq354: mlx5_core1 96698 0 irq355: mlx5_core1 473324 2 irq356: mlx5_core1 86760 0 irq357: mlx5_core1 11590861 52 I expected number of IRQs to be equal number of CPUS. According to Mellanox docs, I should be able to pin each interrupt to specific core to loadbalance. How can I do this in this case when number of IRQs is larger than number of cores? Is there any way to lower the number of interrupts?
You can lower the number of interrupts by changing the coalescing sysctl's in the mce.<N>.conf tree.
dev.mce.0.conf.tx_coalesce_pkts: 32 dev.mce.0.conf.tx_coalesce_usecs: 16 dev.mce.0.conf.rx_coalesce_pkts: 32 dev.mce.0.conf.rx_coalesce_usecs: 3 For example 1024 pkts and 125 us. And also set the queue size bigger than 1024 pkts: dev.mce.0.conf.rx_queue_size: 1024 dev.mce.0.conf.tx_queue_size: 1024 --HPS _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
