This is a bit complicated and I still don't understand it fully, but here
are few pointers. BTW all that follows has nothing specific to memcached.
1) cat /proc/interrupts
This will show the list of which core(s) are handling the interrupts for
given devices.
Check eth0/eth1 whichever you are using. The first column is the irq id.
2) cat /proc/irq/<irq-number>/smp_affinity
If things are good, you should be able to do "echo "xxxx"
> /proc/irq/<irq-number>/smp_affinity to modify the smb affinity. You can
do a bit of googling to find about this.
If this doesn't works, it is either your NIC, or its driver or the kernel.
For our hardware centOS (2.6.18) was not spreading the interrupts but with
Ubuntu (2.6.32) this worked.
I have seen this working with kernel 2.6.24 with Intel NICs.
If nothing else works try setting up multiple memcached instances on the
same machine using different NICs. For example if you have 4 NICs, create 4
memcached instances each with 7/8GB memory. Usually OS will assign different
NICs to different cores (check with cat /proc/interrupts). So let memcached1
listen on eth0, memcached2 on eth1...etc.
Hope this helps.
thanks!
rohitk
On Wed, Sep 29, 2010 at 12:28 AM, Jay Paroline <boxmon...@gmail.com> wrote:
> We've run into this exact same issue and narrowed it down to the NIC,
> but don't really know where to go from there. I'm going to look up
> Dormando's suggestions but if anyone else has experience with this and
> can point us in the right direction, it would be greatly appreciated.
>
> Thanks,
>
> Jay
>
> On Sep 27, 2:34 pm, dormando <dorma...@rydia.net> wrote:
> > > We have an 2 x quad core server with 32 gb ram. If many clients
> > > connect to this server (only memcached runs on it) the first core run
> > > to nearly 100 % use by si (software interrups) and so some client
> > > can't reach the server.
> > > Memcached runs currently with 4 threads and with version (1.4.2). All
> > > other cores have 70 % idle so I ask me is there a possibility to
> > > improve the performance?
> >
> > This is an issue with how your network interrupts are being routed, not
> > with how memcached is being threaded.
> >
> > Wish I had some good links offhand for this, because it's a little
> obscure
> > to deal with. In short; you'll want to balance your network interrupts
> > across cores. Google for blog posts about smp_affinity for network cards
> > and irqbalance (which poorly tries to automatically do this).
> >
> > Depending on how many NIC's you have and if it's multiqueued or not
> you'll
> > have to tune it differently. Linux 2.6.35 has some features for extending
> > the speed of single-queued NIC's (find the pages discussing it on
> > kernelnewbies.org).
>
