Hi Ivan,
you should probably do a monitoring with iostat and vmstat also. On the
top of my head I can think of at least 3 or 4 reasons why this might be
happening. I have similar problems with a simple laptop machine without
LXC containers on it (and don't have such on a server with a bunch of
containers on it). In my experience with bad SI everything always come
back to be RAM related. Also check your filesystem performance. Most of
the FSes nowadays keep a ton of the journalling info in RAM. I know my
response is not exactly an answer to your specific question but I hope
it might give you some pointers for better monitoring of the situation.
BR,
--ilf
On Thu, 2011-09-15 at 09:12 -0600, Ivan Fetch wrote:
> Hello,
>
> I've inherited a Sun 4540 (thumper) machine running 9 LXC containers. During
> the past few weeks we've been troubleshooting a decline in performance, which
> ends up in high %SI (software interrupt) CPU usage. I'm hoping someone here
> can help troubleshoot and narrow down what the real issue is - this one
> really has me stumped.
>
> THis box has 48 disks, 5 RAID6 which are in a RAID0, using md. Two NICs are
> bonded together, and a bridge is used for the box's IP, and the LXC network
> interfaces.
>
> Linux is Ubuntu 10.04, LXC 0.6.3 , containers are also 10.04. Containers run
> Apache, some custom image processing, gaussian, and FTP server...
>
> The box performs well after a reboot, with all containers back online. After
> ~5 days, we notice that the box is sluggish, and backup jobs (Netbackup) get
> less than 1Mb/sec over the network. CPU eventually reaches 61% SI. OTher
> processes (I am looking at ps -ax -o pcpu ..... |sort -n) begin taking much
> higher percent CPU than they should need, I imagine because the high %SI is
> taking cycles; E.G. I'll briefly see ps or sort or a shell using 6% CPU. Top
> shows %sy between 5-20, %wa under 5.
> Memory (32Gb) is mostly used for cache, and there is no swapping.
>
> I know next-to-nothing about tracking down the cause for high %SI CPU usage.
>
>
> Thanks for any help looking at this with a clear head,
>
> - Ivan
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> .
> ------------------------------------------------------------------------------
> Doing More with Less: The Next Generation Virtual Desktop
> What are the key obstacles that have prevented many mid-market businesses
> from deploying virtual desktops? How do next-generation virtual desktops
> provide companies an easier-to-deploy, easier-to-manage and more affordable
> virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
> _______________________________________________
> Lxc-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/lxc-users
------------------------------------------------------------------------------
Doing More with Less: The Next Generation Virtual Desktop
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops? How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
_______________________________________________
Lxc-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lxc-users
