On 1/25/12 7:34 PM, Whitaker, James X. -ND wrote:
works) and have a 512MB ramcache configured.... with 5Gig of RAM allocated to the VM. I shut off all logging, all peer caching... anything that can hit the disks or introduce any kind of dependency, and tell it to serve up a 100 byte chunk of cached data....
Are you fetching a Range: request? Is the object large ? There are known performance problems with fetching ranges of a large objects.
Or are you talking Chunked encoded responses, with no content-length? Once in cache, we generally don't serve chunked encoded (they get transformed to a response with Content-Length: header once it's in the cache, since we now know the length).
rs between X2.0 ms and X4.0ms for each request, where X is a random tens place digit. This suggests a 10ms granularity in a timer or a reference clock. Maybe VMWare or CentOS only supports 10ms timer granularity when something doesn't get immediately pulled off the queue?
10ms sounds like a retry / rescheduling of e.g. a lock. Perhaps try modifying any of
iocore/net/P_UnixNet.h: #define NET_RETRY_DELAY HRTIME_MSECONDS(10) iocore/cluster/P_ClusterCacheInternal.h: #define CACHE_RETRY_PERIOD HRTIME_MSECONDS(10)
There are a few other places where we have 10ms retry times too, but try changing one of the above, and see if you see a difference. Assuming you can reproduce this easily, it ought to help a bit isolating where things are going wrong.
Also, you can try turning on slow-log in records.config, set it at some reasonably long time, and you can hopefully get some ideas where it's taking a long time to process the requests.
I'm rather stumped. Does anyone there have ideas? Is the answer ³ATS was not designed to run on VM's²?
When you run on a bare metal box, do you not see the same problem? I can't say that we've spent any time trying to test (or optimize) for VMs. I've personally seen pretty abysmal performance out of e.g. VirtualBox, but it's the same for both ATS and Varnish for example (typically, I see 1/5th - 1/10th the throughput on a VM vs the bare metal).
Cheers, -- leif
