Zhihui, Where did you disable polling? On sender or receiver? Because like you, I am getting 8.8 Gbps on receiver but if I disable polling on receiver, I drop down to under 2 Gbps!!
So clearly polling on receive side is necessary. Now before we go into the transmit side and effect of polling when LSO is in use, can you verify that the issue you were seeing with mac_rx_srs_poll running 100% even without load is resolved? If not, was that on receiver or sender? I am kind of confused about what are we trying to debug (receiver or sender). Thanks, Sunay PS: I am looking at your other email as well and I can explain why you see the behaviour on transmit and explore ways on fixing it, but I need to clearly understand what we are doing first. zhihui Chen wrote: > Thanks, I have tried your method and the poll function is > disabled. After that, the context switches is decreased very much, but > the performance is still remained at 8.8Gbps. Mpstat output like following: > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl > 0 0 0 0 33 1 15 8 0 1366 0 134084 3 97 > 0 0 > 1 10 0 0 56 16 35208 6 8 1736 0 9 0 48 > 0 52 > 2 4 0 37 19646 19618 58 0 10 408 0 154 0 34 > 0 66 > 3 0 0 0 308 107 118 0 6 1 0 185 0 0 0 100 > > > Then I use the same dtrace command again, the performance is improved to > 9.5Gbps and context switches is also reduced from 35000 to 6300. Mpstat > output likes following: > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl > 0 0 0 0 17 3 1858 8 3 605 0 142000 3 81 > 0 16 > 1 0 0 0 15 6 4472 0 2 2516 0 0 0 92 0 8 > 2 0 0 0 19740 19679 126 0 6 253 0 289 0 46 > 0 54 > 3 0 0 9 509 208 66 0 4 0 0 272 0 0 0 100 > > Because this is TX heavy workload, maybe we should care more about > thread mac_soft_ring_worker??? > > Thanks > Zhihui > > 2009/4/1 Sunay Tripathi <Sunay.Tripathi at sun.com > <mailto:Sunay.Tripathi at sun.com>> > > Sure. Just run this and polling will get disabled > % dladm create-vnic -l ixgbe0 vnic1 > > Let us know what you get with polling disabled. We don't have a > tunable to disable polling but since ixgbe can't assign rx rings to > VNIC yet, it disable polling for primary NIC as well. > > Cheers, > Sunay > > zhihui Chen wrote: > > During my test for 10GBE(Intel Ixgbe) with snv_110, I find the > context switch is a big problem for the performance. Benchmark: > Netperf-2.4.4 > Workload: TCP_STREAM (sending 8KB-size tcp packets from SUT to > remote machine) > > Crossbow use two kernel threads ( mac_soft_ring_worker and > mac_rx_srs_poll_ring) to help send and recv packets in the > kernel. On multi-core or multi-processor system, these two > threads and interrupt for nic can run on different CPU. > Considering following scenario on my 4-core system: > mac_soft_ring_worker: CPU 1 > mac_rx_srs_poll_ring: CPU 1 > Interrupt: CPU 2 > > I run the workload and bind the application to free CPU 0, then > I get the performance results at 8.8Gbps and mpstat output like > following: CPU minf mjf xcal intr ithr csw icsw migr smtx > srw syscl usr sys wt idl > 0 0 0 0 17 3 21 9 0 1093 0 134501 3 > 97 0 0 > 1 0 0 0 29 13 56972 2 7 992 0 2 0 > 50 0 50 > 2 14 0 0 19473 19455 37 0 8 0 0 149 0 > 28 0 72 > 3 0 0 1 305 104 129 0 4 1 0 9 0 > 0 0 100 > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr > sys wt idl > 0 0 0 0 14 2 2 7 0 1120 0 133511 3 > 97 0 0 > 1 0 0 0 24 12 54501 2 6 971 0 2 0 > 48 0 52 > 2 0 0 0 19668 19648 45 0 9 0 0 149 0 > 28 0 72 > 3 0 0 0 306 104 128 0 6 0 0 11 0 > 0 0 100 > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr > sys wt idl > 0 0 0 0 14 2 21 8 2 1107 0 134569 3 > 97 0 0 > 1 0 0 0 32 16 57522 2 6 928 0 2 0 > 50 0 50 > 2 0 0 0 19564 19542 46 0 10 1 0 140 0 > 28 0 72 > 3 0 0 0 306 104 122 0 7 0 0 58 0 > 0 0 100 > > > Next step, I just run one dtrace command: dtrace -n > 'mac_tx:ent...@[probefunc,stack()]=count();}' > Then I can get performance at 9.57Gbps and mpstat output like > following: > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr > sys wt idl > 0 0 0 0 23 5 2055 9 4 529 0 142719 3 > 81 0 15 > 1 0 0 1 21 8 24343 0 2 2523 0 0 0 > 88 0 12 > 2 14 0 5 19678 19537 81 0 5 0 0 150 0 > 43 0 57 > 3 0 0 6 308 104 93 0 5 2 0 278 0 > 0 0 100 > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr > sys wt idl > 0 0 0 2 19 4 1998 9 6 556 0 142911 3 > 82 0 16 > 1 0 0 0 20 8 23543 1 2 2556 0 0 0 > 88 0 12 > 2 0 0 6 19647 19499 106 0 8 1 0 266 0 > 43 0 57 > 3 0 0 2 308 104 70 0 5 1 0 28 0 > 0 0 100 > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr > sys wt idl > 0 0 0 0 21 3 1968 10 4 556 0 144547 3 > 82 0 15 > 1 0 0 0 20 10 23334 0 2 2622 0 0 0 > 90 0 10 > 2 0 0 9 19797 19658 92 0 10 1 0 274 0 > 44 0 56 > 3 0 0 0 307 104 95 0 6 2 0 182 0 > 0 0 99 > > I dont think dtrace can help improve the performance of nic. If > you compare the mpstat output, the biggest difference is that > context switch has been reduced very much from 55000 to 23000. > This leads to my point that two much context switches hinders > the performance of crossbow. > If I make these two kernel threads and interrupt run in totally > different cores, the performance can be reduced to about 7.8Gbps > while context switches will be increase to at about 80000 per > second, but the interrupts remains at about 19500/s > > In crossbow, thread mac_soft_ring_worker will be wakeup by > mac_rx_srs_poll_ring and interrupt through calling the function > mac_soft_ring_worker_wakeup. I just think that if I can disable > the polling function, then context switches should be reduced. > Thanks > Zhihui > > > > 2009/4/1 rajagopal kunhappan <rajagopal.kunhappan at sun.com > <mailto:rajagopal.kunhappan at sun.com> > <mailto:rajagopal.kunhappan at sun.com > <mailto:rajagopal.kunhappan at sun.com>>> > > > Hi Zhihui, > > > In crossbow, each mac_srs has a kernel thread called > "mac_rx_srs_poll_ring" > to poll the hardware and crossbow will wakeup this thread to > poll packets > from the hardware automatically. Does crossbow provide any > method to disable > the polling mechanism, for example disabling the this > kernel thread? > > > Presently no. Can we know why you would want to do that? > > Thanks, > -krgopi > > Thanks > Zhihui > >
