Your SDC is overloaded
is your filesystem mounted noatime?
Artur
On Apr 9, 2009, at 2:46 PM, Cloude Porteus wrote:
The current load is just above 2. I'll check this again when I see a
load spike.
[clo...@squid03 ~]$ iostat -k -x 5
Linux 2.6.18-53.1.19.el5.centos.plus
(squid03.instructables.com) 04/09/2009
avg-cpu: %user %nice %system %iowait %steal %idle
1.19 0.00 0.95 2.14 0.00 95.73
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sda 0.07 9.64 0.15 1.65 10.08 45.68
61.80 0.13 70.32 3.96 0.72
sdb 0.07 9.63 0.15 1.66 10.14 45.68
61.75 0.02 10.03 3.76 0.68
sdc 0.03 16.47 1.21 14.69 13.99 128.81
17.96 0.08 4.81 4.31 6.85
sdd 0.03 16.45 1.17 13.24 13.29 119.96
18.49 0.24 16.52 4.06 5.86
md1 0.00 0.00 0.43 11.13 20.19 44.52
11.19 0.00 0.00 0.00 0.00
md2 0.00 0.00 2.41 29.40 26.58 117.61
9.07 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00
3.15 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 2.40 46.70 0.00 50.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.40 6.00 238.40 74.40 974.40
8.58 132.88 515.03 4.09 100.02
sdd 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 1.80 67.67 0.00 29.63
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdc 0.00 1.60 13.40 141.80 188.80 1053.60
16.01 138.62 934.04 6.44 100.02
sdd 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.50 0.00 1.80 61.40 0.00 36.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.40 0.00 2.40
12.00 0.00 9.00 9.00 0.36
sdb 0.00 0.00 0.00 0.40 0.00 2.40
12.00 0.00 9.50 9.50 0.38
sdc 0.00 1.60 6.40 257.00 132.00 2195.20
17.67 107.40 450.21 3.68 96.82
sdd 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.20 0.00 0.80
8.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.60 0.00 1.60 47.80 0.00 50.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.20 0.00 1.60
16.00 0.00 11.00 11.00 0.22
sdb 0.00 0.00 0.00 0.20 0.00 1.60
16.00 0.00 13.00 13.00 0.26
sdc 0.00 0.80 0.20 301.80 8.80 1270.40
8.47 119.40 373.98 3.31 100.04
sdd 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.60 0.00 1.70 47.80 0.00 49.90
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdc 0.00 1.20 2.40 245.31 43.11 1538.52
12.77 101.41 419.12 4.03 99.80
sdd 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.60 0.00 1.50 3.20 0.00 94.69
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdb 0.20 0.00 0.40 0.00 2.40 0.00
12.00 0.01 14.00 7.00 0.28
sdc 0.00 0.00 6.60 11.00 174.40 192.80
41.73 1.26 421.34 3.73 6.56
sdd 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.70 0.00 1.60 29.50 0.00 68.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 5.60 208.60 110.40 857.60
9.04 70.18 301.18 2.90 62.06
sdd 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.50 0.00 1.50 48.05 0.00 49.95
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sda 0.00 0.20 0.00 0.80 0.00 5.60
14.00 0.01 8.75 8.75 0.70
sdb 0.00 0.20 0.00 0.80 0.00 5.60
14.00 0.01 9.50 9.50 0.76
sdc 0.00 1.00 6.80 232.40 91.20 1180.80
10.64 110.32 475.49 4.18 100.02
sdd 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.60 0.00 2.40
8.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
On Thu, Apr 9, 2009 at 1:43 PM, Artur Bergman <s...@crucially.net>
wrote:
What is your iopressure?
iostat -k -x 5
or something like that
artur
On Apr 9, 2009, at 12:27 PM, Cloude Porteus wrote:
Varnishstat doesn't list any nuked objects and file storage and
shmlog look like they have plenty of space:
df -h
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Filesystem Size Used Avail Use% Mounted on
tmpfs 150M 81M 70M 54% /usr/local/var/varnish
/dev/sdc1 74G 11G 61G 16% /var/lib/varnish
top
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
top - 12:26:33 up 164 days, 22:21, 1 user, load average: 2.60,
3.26, 3.75
Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.3%sy, 0.0%ni, 97.0%id, 0.7%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 8183492k total, 7763100k used, 420392k free, 13424k
buffers
Swap: 3148720k total, 56636k used, 3092084k free, 7317692k
cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7441 varnish 15 0 70.0g 6.4g 6.1g S 2 82.5 56:33.31 varnishd
Varnishstat:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Hitrate ratio: 8 8 8
Hitrate avg: 0.9782 0.9782 0.9782
36494404 219.98 160.57 Client connections accepted
36494486 220.98 160.57 Client requests received
35028477 212.98 154.12 Cache hits
474091 4.00 2.09 Cache hits for pass
988013 6.00 4.35 Cache misses
1465955 10.00 6.45 Backend connections success
9 0.00 0.00 Backend connections failures
994 . . N struct sess_mem
11 . . N struct sess
274047 . . N struct object
252063 . . N struct objecthead
609018 . . N struct smf
28720 . . N small free smf
2 . . N large free smf
2 . . N struct vbe_conn
901 . . N struct bereq
2000 . . N worker threads
2000 0.00 0.01 N worker threads created
143 0.00 0.00 N overflowed work requests
1 . . N backends
672670 . . N expired objects
3514467 . . N LRU moved objects
49 0.00 0.00 HTTP header overflows
32124238 206.98 141.34 Objects sent with write
36494396 224.98 160.57 Total Sessions
36494484 224.98 160.57 Total Requests
783 0.00 0.00 Total pipe
518770 4.00 2.28 Total pass
1464570 10.00 6.44 Total fetch
14559014884 93563.69 64058.18 Total header bytes
168823109304 489874.04 742804.45 Total body bytes
36494387 224.98 160.57 Session Closed
203 0.00 0.00 Session herd
1736767745 10880.80 7641.60 SHM records
148079555 908.90 651.53 SHM writes
15088 0.00 0.07 SHM flushes due to overflow
10494 0.00 0.05 SHM MTX contention
687 0.00 0.00 SHM cycles through buffer
2988576 21.00 13.15 allocator requests
580296 . . outstanding allocations
8916353024 . . bytes allocated
44770738176 . . bytes free
656 0.00 0.00 SMS allocator requests
303864 . . SMS bytes allocated
303864 . . SMS bytes freed
1465172 10.00 6.45 Backend requests made
On Thu, Apr 9, 2009 at 12:18 PM, Artur Bergman <s...@crucially.net>
wrote:
For the file storage or for the shmlog?
When do you start nuking/expiring from disk? I suspect the load
goes up when you run out of storage space?
Cheers
Artur
On Apr 9, 2009, at 12:02 PM, Cloude Porteus wrote:
Has anyone experienced very high server load averages? We're
running varnish on a dual core with 8gb of ram. It runs okay for a
day or two and then I start seeing load averages in 6-10 range for
an hour or so, drops down to 2-3, then goes back up.
This starts to happen once we have more items in the cache than
our physical memory. Maybe increasing our lru_interval will help?
It's currently set to 3600.
Right now we're running with a 50gb file storage option. There are
270k objects in the cache, 70gb virtual memory, 6.2gb of res
memory used, 11gb of data on disk in the file storage. We have a
98% hit ratio.
We followed Artur's advice about setting a tmpfs and creating an
ext2 partition for our file storage.
I also tried running with malloc as our storage type, but I had to
set it at a little less than half of our physical ram in order for
it to work well after the cache got full. I don't understand why
the virtual memory is double when I am running in malloc mode. I
was running it with 5gb and the virtual memory was about 10-12gb
and once it got full it started using the swap memory.
Thanks for any help/insight.
best,
cloude
--
VP of Product Development
Instructables.com
http://www.instructables.com/member/lebowski
_______________________________________________
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
--
VP of Product Development
Instructables.com
http://www.instructables.com/member/lebowski
--
VP of Product Development
Instructables.com
http://www.instructables.com/member/lebowski
_______________________________________________
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev