Garrett Moore wrote:
I've been watching my memory usage and I have no idea what is consuming
memory as 'Active'.
Last night I had around 6500MB 'Active' again, 1500MB Wired, no inact, ~30MB
buf, no free, and ~100MB swap used. My performance copying ZFS->ZFS was
again slow (<1MB/s). I tried killing rTorrent and no significant amount of
memory was reclaimed - maybe 100MB. `ps aux` showed no processes using any
significant amount of memory, and I was definitely nowhere near 6500MB
usage.
I tried running the perl oneliner again to hog a bunch of memory, and almost
all of the Active memory was IMMEDIATELY marked as Free, and my performance
was excellent again.
I'm not sure what in userland could be causing the issue. The only things
I've installed are rTorrent, lighttpd, samba, smartmontools, vim, bash,
Python, Perl, and SABNZBd. There is nothing that *should* be consuming any
serious amount of memory.
Last night I tried ZFS with pool on iSCSI connected Dell MD3000i and I
was suprised by too low speed of simple cp -a command (copying from UFS
partition to ZFS) The write speed was about 2MB/s only.
After looking in to ARC stuff, I realized some weird values:
ARC Size:
Current Size: 1 MB (arcsize)
Target Size (Adaptive): 205 MB (c)
Min Size (Hard Limit): 205 MB (zfs_arc_min)
Max Size (Hard Limit): 1647 MB (zfs_arc_max)
(stats from script http://cuddletech.com/arc_summary/
freebsd version
http://bitbucket.org/koie/arc_summary/changeset/dbe14d2cf52b/ )
I don't know why it shows Current Size 1MB.
I tried the perl oneliner from this thread. Then I got about 5GB free
merory and Target Size and Current Size growed to 1647MB. Write speed
increase to about 8MB/s and after few minuts slowly dropped to 2MB/s and
ARC Current Size dropped to 1MB again.
This server is not in production and was idle. Just copying the data
from one partition to another.
Today I tried serving the data by Lighttpd.
There is impressive iSCSI read performance - because of ZFS prefetch, it
can achieve 880Mbits of read from iSCSI, but serving by Lighttpd only
about 66Mbits
bce0 - internet
bce1 - iSCSI to storage MD3000i
bce0 bce1
Kbps in Kbps out Kbps in Kbps out
2423.22 65481.56 855970.7 4348.73
2355.26 63911.74 820561.3 4846.08
2424.87 65998.62 848937.1 4312.37
2442.78 66544.95 858019.0 4356.64
iostat -x
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da1 1596.8 3.6 102196.7 22.2 13 7.4 97
da1 1650.2 2.9 105612.7 55.7 16 7.4 103
da1 1647.3 0.0 105422.9 0.0 13 7.2 100
da1 1636.5 2.3 104735.4 20.0 16 7.3 100
da1 1642.9 0.0 105141.1 0.0 13 7.3 100
~/bin/arcstat.pl -f
Time,read,hits,Hit%,miss,miss%,dmis,dm%,mmis,mm%,arcsz,c 30
Time read hits Hit% miss miss% dmis dm% mmis mm% arcsz c
12:18:05 16K 15K 94 838 5 570 3 1 0 16933296
215902720
12:18:36 16K 15K 94 839 5 571 3 0 0 21488288
215902720
12:19:06 16K 15K 94 836 5 569 3 1 0 17228688
215902720
12:19:37 16K 15K 94 839 5 572 3 4 1 22002672
215902720
12:20:07 16K 15K 94 841 5 570 3 1 0 27784960
215902720
12:20:38 16K 15K 94 838 5 569 3 0 0 21839472
215902720
12:21:08 16K 15K 94 837 5 568 3 0 0 28244992
215902720
12:21:39 16K 15K 94 833 5 565 3 1 0 28744416
215902720
12:22:09 16K 15K 94 842 5 576 3 4 1 28646656
215902720
12:22:39 16K 15K 94 840 5 575 3 3 0 28903696
215902720
12:23:10 15K 15K 94 821 5 561 3 0 0 28765904
215902720
12:23:40 16K 15K 94 828 5 566 3 0 0 28395840
215902720
12:24:11 16K 15K 94 828 5 568 3 0 0 32063408
215902720
12:24:41 16K 15K 94 834 5 570 3 0 0 29800976
215902720
12:25:12 15K 15K 94 820 5 562 3 1 0 29066512
215902720
# ~/bin/arc_summary.pl
System Memory:
Physical RAM: 8169 MB
Free Memory : 0 MB
ARC Size:
Current Size: 22 MB (arcsize)
Target Size (Adaptive): 205 MB (c)
Min Size (Hard Limit): 205 MB (zfs_arc_min)
Max Size (Hard Limit): 1647 MB (zfs_arc_max)
ARC Size Breakdown:
Most Recently Used Cache Size: 5% 11 MB (p)
Most Frequently Used Cache Size: 94% 194 MB (c-p)
ARC Efficency:
Cache Access Total: 81843958
Cache Hit Ratio: 95% 78525502 [Defined State
for buffer]
Cache Miss Ratio: 4% 3318456 [Undefined
State for Buffer]
REAL Hit Ratio: 95% 78418994 [MRU/MFU Hits Only]
Data Demand Efficiency: 97%
Data Prefetch Efficiency: 10%
CACHE HITS BY CACHE LIST:
Anon: --% Counter Rolled.
Most Recently Used: 2% 2209869 (mru)
[ Return Customer ]
Most Frequently Used: 97% 76209125 (mfu)
[ Frequent Customer ]
Most Recently Used Ghost: 1% 965711 (mru_ghost)
[ Return Customer Evicted, Now Back ]
Most Frequently Used Ghost: 0% 176871 (mfu_ghost)
[ Frequent Customer Evicted, Now Back ]
CACHE HITS BY DATA TYPE:
Demand Data: 97% 76770304
Prefetch Data: 0% 126644
Demand Metadata: 2% 1628528
Prefetch Metadata: 0% 26
CACHE MISSES BY DATA TYPE:
Demand Data: 63% 2122089
Prefetch Data: 32% 1063449
Demand Metadata: 4% 132894
Prefetch Metadata: 0% 24
---------------------------------------------
# sysctl kstat.zfs.misc.arcstats
kstat.zfs.misc.arcstats.hits: 75409326
kstat.zfs.misc.arcstats.misses: 3144748
kstat.zfs.misc.arcstats.demand_data_hits: 73731356
kstat.zfs.misc.arcstats.demand_data_misses: 2003526
kstat.zfs.misc.arcstats.demand_metadata_hits: 1551917
kstat.zfs.misc.arcstats.demand_metadata_misses: 132730
kstat.zfs.misc.arcstats.prefetch_data_hits: 126027
kstat.zfs.misc.arcstats.prefetch_data_misses: 1008468
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 26
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 24
kstat.zfs.misc.arcstats.mru_hits: 2105758
kstat.zfs.misc.arcstats.mru_ghost_hits: 914887
kstat.zfs.misc.arcstats.mfu_hits: 73197609
kstat.zfs.misc.arcstats.mfu_ghost_hits: 171171
kstat.zfs.misc.arcstats.deleted: 2367973
kstat.zfs.misc.arcstats.recycle_miss: 412788
kstat.zfs.misc.arcstats.mutex_miss: 2865
kstat.zfs.misc.arcstats.evict_skip: 17459
kstat.zfs.misc.arcstats.hash_elements: 2478
kstat.zfs.misc.arcstats.hash_elements_max: 28921
kstat.zfs.misc.arcstats.hash_collisions: 86135
kstat.zfs.misc.arcstats.hash_chains: 25
kstat.zfs.misc.arcstats.hash_chain_max: 3
kstat.zfs.misc.arcstats.p: 14908416
kstat.zfs.misc.arcstats.c: 215902720
kstat.zfs.misc.arcstats.c_min: 215902720
kstat.zfs.misc.arcstats.c_max: 1727221760
kstat.zfs.misc.arcstats.size: 30430560
kstat.zfs.misc.arcstats.hdr_size: 555072
kstat.zfs.misc.arcstats.l2_hits: 0
kstat.zfs.misc.arcstats.l2_misses: 0
kstat.zfs.misc.arcstats.l2_feeds: 0
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_writes_sent: 0
kstat.zfs.misc.arcstats.l2_writes_done: 0
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_free_on_write: 0
kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_size: 0
kstat.zfs.misc.arcstats.l2_hdr_size: 0
kstat.zfs.misc.arcstats.memory_throttle_count: 135489
This is on FreeBSD 7.2-STABLE #0: Sun Dec 6 23:21:17 CET 2009
r...@dust.hrej.cz:/usr/obj/usr/src/sys/GENERIC amd64
Can somebody tell me, why ARC Current Size is dropping too low? (1-20MB
if arc_min is 205MB)
The system have 8GB of memory and 8 CPU cores:
last pid: 83605; load averages: 0.17, 0.15, 0.10 up 36+10:34:34
12:29:05
58 processes: 1 running, 56 sleeping, 1 zombie
CPU: 0.1% user, 0.0% nice, 2.3% system, 1.7% interrupt, 95.8% idle
Mem: 237M Active, 6259M Inact, 1154M Wired, 138M Cache, 827M Buf, 117M Free
Swap: 8192M Total, 96K Used, 8192M Free
I have no loader.conf tunning on this machine.
Miroslav Lachman
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
