Shreyas wrote:
Hello all,
I face a similar problem of un-utilized CPUs.
I have a Intel Xeon 2.66GHz, with 8 CPU cores. The machine has Opensolaris
build 48. I am using a 10 Gig card, whose driver I have written, to run iperf
at full blast.
I see that utilization of one of the CPUs is 80-100% where as rest of them stay
idle at 0-3 %. Earlier I thought, this was due to locking in my driver code. I
modified the driver code to reduce time under locking to a great extent. I used
'lockstat -o output_file sleep 20' to verify this. But this did not improve
the situation atall. The mpstat looks like following :
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 1388 220 102 0 2 52 0 79 0 3 0 97
1 13 0 0 1079 0 76 0 1 7 0 111 0 1 0 99
2 0 0 0 1042 0 4 0 2 0 0 0 0 1 0 99
3 0 0 0 1054 4 0 0 0 0 0 0 0 0 0 100
4 0 0 6146 207 0 70 0 1 316 0 10 0 2 0 98
5 0 0 0 1068 3 30 0 2 0 0 0 0 0 0 100
6 0 0 1036 917 2 58 1 2 476 0 65 0 6 0 94
7 0 0 161 62340 61308 6363 2139 1 2442 0 530 0 99 0 1
CPU(7) has 99% utilization. The throughput of iperf increases greatly when more
CPUs are utilized.
Can I somehow distribute the interrupt handling among all the CPUs available ?
I have even used softirq (deffered work) in which I complete most of the
interrupt handling work. Can I attach the softirq routine to a CPU when I
trigger it ? Or can I tweak some system parameters ? The intrd distribution of
interrupts does not seem to work with me :(
/var/adm/messages show this message :
Dec 27 19:16:12 opensol-48 unix: [ID 954099 kern.info] NOTICE: IRQ16 is being
shared by drivers with different interrupt levels.
Dec 27 19:16:12 opensol-48 This may result in reduced system performance.
I have tried to change the interrupt level of my network driver interrupt
handler, but ddi_intr_set_pri() returns failure for any level other than 6. I
am using legacy interrupts as I could not initialize MSI interrupts. Will MSI/X
interrupts help avoid this sharing of interrupt line ?
Do I need to do anything in driver to take advantage of soft rings from dl ?
(it seems I do not need to do anything)
If your driver is a nemo (gldv3) driver, then setting ip_squeue_fanout
to 1 will use soft rings.
By default, 2 soft rings are created. You can create more by setting
ip_soft_rings_cnt.
Crossbow project (http://opensolaris.org/os/project/crossbow/) will be
providing additional
features that will help 10Gbps NICs.
Among them features like 1) specifying the CPUs that a NIC should use
for processing
incoming and outgoing packets. This would cause the interrupt to be
assigned to one of the
specified CPU and use other specified CPUs to process packets via soft
ring worker threads.
2) polling and blanking mechanisms. This would ask the NIC to stop
interrupting, and in stead
use worker threads to pull packets from the NICs.
Again these are part of Crossbow still under development. Early bits are
available from
the project page.
-krgopi
Thanks
Shreyas
This message posted from opensolaris.org
_______________________________________________
networking-discuss mailing list
[email protected]
_______________________________________________
networking-discuss mailing list
[email protected]
