We are running a web workload on system with 8 cores, each core has two
hyper threads. The workload is network intensive. During test we find that
the system is very busy in kernel and has very high cross call rates. Dtrace
shows that most xcalls are caused by memory free in kernel. Any suggestion
to lower sys utilization and reduce xcalls?
"mpstat -a 2" output:
SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
sze
0 805 14 322274 348634 17344 50496 7537 3396 14842 185 59338 2 18
0 80 16
SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
sze
0 11027 1 2748955 2864793 89324 148049 30088 5111 123434 22 281361
7 92 0 0 16
SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
sze
0 10304 0 2772163 2890354 90325 155787 31692 5456 108526 12 262906
6 94 0 0 16
SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
sze
0 18048 0 2776165 2891182 88630 152517 29913 5077 100318 19 247422
6 94 0 0 16
SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
sze
0 11260 0 2782578 2902024 91238 160978 31941 5604 125821 24 279300
6 94 0 0 16
SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
sze
0 7861 0 2733062 2864302 96346 193671 38932 6320 168122 12 313714 7
93 0 0 16
SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
sze
0 7633 0 2783807 2902558 90803 166278 32752 6122 126578 10 273982 6
94 0 0 16
"nicstat 2" output:
Time Int rKB/s wKB/s rPk/s wPk/s rAvs wAvs %Util
Sat
05:01:55 lo0 0.00 0.00 2179.7 2179.7 0.00 0.00 0.00
0.00
05:01:55 igb0 0.06 0.33 1.00 1.00 64.00 338.0 0.00
0.00
05:01:55 igb1 3825.5 686.5 4424.8 4554.2 885.3 154.4 3.70
0.00
05:01:55 ixgbe2 12399.1 446438 145339 340749 87.36 1341.6 37.6
0.00
05:01:55 ixgbe4 10792.8 434197 128687 332963 85.88 1335.3 36.5
0.00
Time Int rKB/s wKB/s rPk/s wPk/s rAvs wAvs %Util
Sat
05:01:57 lo0 0.00 0.00 2258.3 2258.3 0.00 0.00 0.00
0.00
05:01:57 igb0 0.06 0.29 1.00 1.00 64.00 298.0 0.00
0.00
05:01:57 igb1 3830.8 690.9 4497.1 4588.1 872.3 154.2 3.70
0.00
05:01:57 ixgbe2 12596.9 438299 148012 334978 87.15 1339.8 36.9
0.00
05:01:57 ixgbe4 11330.4 459088 136182 349905 85.20 1343.5 38.5
46.00
Time Int rKB/s wKB/s rPk/s wPk/s rAvs wAvs %Util
Sat
05:01:59 lo0 0.00 0.00 2168.9 2168.9 0.00 0.00 0.00
0.00
05:01:59 igb0 0.06 0.29 1.00 1.00 64.00 298.0 0.00
0.00
05:01:59 igb1 3805.1 690.9 4472.3 4590.2 871.2 154.1 3.68
0.00
05:01:59 ixgbe2 12303.5 422444 144187 324561 87.38 1332.8 35.6
0.00
05:01:59 ixgbe4 10944.4 454153 129900 349436 86.27 1330.9 38.1
4.50
"vmstat 2" output
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s2 s3 s4 s5 in sy cs us sy
id
14 0 0 36683132 29407588 1101 5602 559 1 1 0 3487 35 -0 1171 -0 919649
98705 90237 2 30 67
79 0 0 31365632 23477712 1 9242 0 0 0 0 0 0 0 2169 0 3027764 159718 68502
3 97 0
80 0 0 31293276 23373552 0 5452 0 0 0 0 0 0 0 2291 0 3020344 172791 73594
3 97 0
55 0 0 31067468 23217816 0 5946 0 0 0 0 0 0 0 2164 0 3029687 201127 87720
3 97 0
86 0 0 30961960 23115928 0 5676 0 0 0 0 0 0 0 2253 0 3025300 154396 71515
3 97 0
56 0 0 30798796 22959692 0 6597 0 0 0 0 0 0 0 2340 0 3026345 161389 56838
3 96 0
81 0 0 30796128 22963332 0 6638 0 0 0 0 0 0 0 2396 0 3025128 172106 68770
3 97 0
dtrace -n 'xc_common:xcal...@[stack()]=count();}' -c "sleep 10" output:
.....
unix`xc_do_call+0x135
unix`xc_call+0x4b
unix`hat_tlb_inval+0x2af
unix`x86pte_inval+0xaa
unix`hat_pte_unmap+0xfd
unix`hat_unload_callback+0x193
unix`hat_unload+0x41
unix`segkmem_free_vn+0x6f
unix`segkmem_free+0x27
genunix`vmem_xfree+0x104
genunix`vmem_free+0x29
genunix`kmem_free+0x44
genunix`dblk_lastfree_oversize+0x69
genunix`dblk_decref+0x64
genunix`freemsg+0x84
ixgbe`ixgbe_free_tcb+0x43
ixgbe`ixgbe_tx_recycle_head_wb+0x1d4
ixgbe`ixgbe_intr_rx_tx+0x101
unix`av_dispatch_autovect+0x7c
unix`dispatch_hardint+0x33
29517
unix`xc_do_call+0x135
unix`xc_call+0x4b
unix`hat_tlb_inval+0x2af
unix`x86pte_inval+0xaa
unix`hat_pte_unmap+0xfd
unix`hat_unload_callback+0x193
unix`hat_unload+0x41
unix`segkmem_free_vn+0x6f
unix`segkmem_free+0x27
genunix`vmem_xfree+0x104
genunix`vmem_free+0x29
genunix`kmem_free+0x44
genunix`dblk_lastfree_oversize+0x69
genunix`dblk_decref+0x64
genunix`freeb+0x80
ip`tcp_rput_data+0x24ac
ip`squeue_drain+0x179
ip`squeue_enter+0x3f4
ip`tcp_sendmsg+0xfb
sockfs`so_sendmsg+0x1c7
45907
unix`xc_do_call+0x135
unix`xc_call+0x4b
unix`hat_tlb_inval+0x2af
unix`x86pte_inval+0xaa
unix`hat_pte_unmap+0xfd
unix`hat_unload_callback+0x193
unix`hat_unload+0x41
unix`segkmem_free_vn+0x6f
unix`segkmem_zio_free+0x27
genunix`vmem_xfree+0x104
genunix`vmem_free+0x29
genunix`kmem_slab_destroy+0x87
genunix`kmem_slab_free+0x2bb
genunix`kmem_magazine_destroy+0x39a
genunix`kmem_depot_ws_reap+0x65
genunix`taskq_thread+0x193
unix`thread_start+0x8
541781
unix`xc_do_call+0x135
unix`xc_call+0x4b
unix`hat_tlb_inval+0x2af
unix`x86pte_inval+0xaa
unix`hat_pte_unmap+0xfd
unix`hat_unload_callback+0x193
unix`hat_unload+0x41
unix`segkmem_free_vn+0x6f
unix`segkmem_free+0x27
genunix`vmem_xfree+0x104
genunix`vmem_free+0x29
genunix`kmem_free+0x44
genunix`dblk_lastfree_oversize+0x69
genunix`dblk_decref+0x64
genunix`freeb+0x80
ip`tcp_rput_data+0x24ac
ip`squeue_drain+0x179
ip`squeue_enter+0x3f4
ip`ip_input+0xa8e
mac`mac_rx_soft_ring_drain+0xdf
1118445
unix`xc_do_call+0x135
unix`xc_call+0x4b
unix`hat_tlb_inval+0x2af
unix`x86pte_inval+0xaa
unix`hat_pte_unmap+0xfd
unix`hat_unload_callback+0x193
unix`hat_unload+0x41
unix`segkmem_free_vn+0x6f
unix`segkmem_free+0x27
genunix`vmem_xfree+0x104
genunix`vmem_free+0x29
genunix`kmem_free+0x44
genunix`dblk_lastfree_oversize+0x69
genunix`dblk_decref+0x64
genunix`freeb+0x80
ip`tcp_rput_data+0x24ac
ip`squeue_drain+0x179
ip`squeue_polling_thread+0x1dd
unix`thread_start+0x8
1919538
unix`xc_do_call+0x135
unix`xc_call+0x4b
unix`hat_tlb_inval+0x2af
unix`x86pte_inval+0xaa
unix`hat_pte_unmap+0xfd
unix`hat_unload_callback+0x193
unix`hat_unload+0x41
unix`segkmem_free_vn+0x6f
unix`segkmem_free+0x27
genunix`vmem_xfree+0x104
genunix`vmem_free+0x29
genunix`kmem_free+0x44
genunix`dblk_lastfree_oversize+0x69
genunix`dblk_decref+0x64
genunix`freeb+0x80
ip`tcp_rput_data+0x24ac
ip`squeue_drain+0x179
ip`squeue_enter+0x3f4
ip`ip_input+0xc17
mac`mac_rx_soft_ring_drain+0xdf
3678526
Thanks
Zhihui
_______________________________________________
perf-discuss mailing list
[email protected]
