Hi All, After much investigation and help from our ops department, it was found that the ip_conntrack buffer on the host OS was filling up under load tests.
Disabling iptables on the host (/etc/init.d/iptables stop) resolved the issue and did not seem to impact the hosted VMs. Regards, Noel On Monday 16 March 2009 12:32:54 sebb wrote: > On 16/03/2009, Noel O'Brien <nobr...@newbay.com> wrote: > > On Monday 16 March 2009 12:01:53 sebb wrote: > > > On 16/03/2009, Noel O'Brien <nobr...@newbay.com> wrote: > > > > Hi All, > > > > > > > > > > > > > > > > I've been running some load testing in a virtual server (Xen) as > > > > that's > > > > > > > > where our product will be deployed. I've noticed some unusual > > > > behavior > > > > > > > > and I can't seem to be able to pinpoint what's going wrong. > > > > > > > > > > > > > > > > Basically, I start 1 JMeter instance running 10 threads against the > > > > > > > > product (a http REST server in Tomcat). It connects to several > > > > auxillary > > > > > > > > programs (DB, other HTTP servers, SwiftMQ, etc), all of which are on > > > > > > > > different virtual machines within the same subnet. I'm using top and > > > > lsof > > > > > > to monitor the resources of each running application. Virtualised OS > > > > is > > > > > > > > RHEL 5, running Tomcat 6.0.16 and Java 1.6.0_07 > > > > > > > > > > > > > > > > The problem is that every few mins (about 3) the CPU usage of all the > > > > > > > > processes I'm monitoring (product, JMeter, DB, etc.) all drop to 0% > > > > for > > > > > > > > about 30 seconds. None of the logs suggest that anything is wrong, so > > > > > > > > it's incredibly hard to track down what's causing this. I'm thinking > > > > that > > > > > > this issue is caused by the virtual servers upon which the programs > > > > are > > > > > > > > running, perhaps mis- / mal-configuration of the virtual network > > > > cards. > > > > > > > > > > > > > > > > Has anyone else experienced anything like this while stress testing > > > > in a > > > > > > > > virtualised environment? > > > > > > What happens to I/O during this time? > > > > I'm not sure. lsof blocks, so running it during this time is fruitless. > > Is there another command I can run to check I/O? > > Doesn't top show I/O? > > Also, try running ping to each of the systems (ideally from a non-Xen > system). > > This should help show if it is a single system locking up and causing > the others to wait for it. > > Are all the Xen servers on the same host? > > > > Do top and lsof continue responding OK? > > > > Top continues to respond (unless it blocks and shows 0% CPU for the > > duration), but lsof blocks until the CPU appears to be given back to the > > processes. > > > > > > Regards, > > > > > > > > > > > > > > > > Noel > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: > > > > jmeter-user-unsubscr...@jakarta.apache.org > > > > > For additional commands, e-mail: > > > > jmeter-user-h...@jakarta.apache.org
