Dear FreeBSD community,
We're facing a somewhat odd issue, perhaps similar to what is discussed
here: https://forums.freebsd.org/threads/l2arc-degraded.47540/
The issue is that the L2ARC header seems to grow without limit, similar
to a memory leak, pressuring more and more memory over time out of the ARC.
For example, the output of "zpool iostat -v 1"
capacity operations bandwidth
pool alloc free read write read write
------------ ----- ----- ----- ----- ----- -----
syspool 1.15G 275G 0 0 0 0
mirror 1.15G 275G 0 0 0 0
gpt/zfs0 - - 0 0 0 0
gpt/zfs1 - - 0 0 0 0
------------ ----- ----- ----- ----- ----- -----
tank 1.21T 1.51T 229 1.99K 3.67M 9.48M
mirror 124G 154G 67 125 787K 503K
da0 - - 20 27 440K 503K
da1 - - 45 28 379K 503K
[...]
mirror 124G 154G 34 164 454K 612K
da18 - - 26 12 417K 612K
da19 - - 6 13 58.8K 612K
logs - - - - - -
mirror 117M 74.4G 0 109 0 1.75M
da21 - - 0 109 0 1.75M
da22 - - 0 109 0 1.75M
cache - - - - - -
da23 1.67T 16.0E 302 7 2.85M 223K
------------ ----- ----- ----- ----- ----- -----
Here the cache shows 1.67T, in use and 16.0E free.
The cache is a 373GB Intel SSD.
# diskinfo -v da23
da23
512 # sectorsize
400088457216 # mediasize in bytes (373G)
781422768 # mediasize in sectors
4096 # stripesize
0 # stripeoffset
48641 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
BTTV4234089C400HGN # Disk ident.
id1,enc@n500e004aaaaaaa3e/type@0/slot@18 # Physical path
The L2ARC stats section from "zfs-stats -a":
L2 ARC Summary: (DEGRADED)
Passed Headroom: 133.33m
Tried Lock Failures: 4.90b
IO In Progress: 313.63k
Low Memory Aborts: 1.52k
Free on Write: 589.79k
Writes While Full: 34.57k
R/W Clashes: 46.95k
Bad Checksums: 408.40m
IO Errors: 151.99m
SPA Mismatch: 632.00m
L2 ARC Size: (Adaptive) 1.89 TiB
Header Size: 0.88% 16.98 GiB
L2 ARC Evicts:
Lock Retries: 1.27k
Upon Reading: 2
L2 ARC Breakdown: 2.10b
Hit Ratio: 32.89% 691.15m
Miss Ratio: 67.11% 1.41b
Feeds: 3.70m
L2 ARC Buffer:
Bytes Scanned: 10.70 PiB
Buffer Iterations: 3.70m
List Iterations: 236.30m
NULL List Iterations: 24.86m
L2 ARC Writes:
Writes Sent: 100.00% 3.38m
Here we can see that currently the Header Size is almost 17gb.
This header size grows continuously without (apparent) limit.
Also zfs appears to think it's holding 1.89 TiB inside the L2ARC, which
seems very very unlikely.
# freebsd-version
10.1-RELEASE-p13
# uname -a
FreeBSD servername 10.1-RELEASE-p10 FreeBSD 10.1-RELEASE-p10 #0: Wed May
13 06:54:13 UTC 2015
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
# uptime
4:35PM up 42 days, 15:24, 1 user, load averages: 1.35, 0.96, 0.84
Does anyone know how we can alleviate the issue?
We originally thought the issue was caused by
https://www.freebsd.org/security/advisories/FreeBSD-EN-15:07.zfs.asc
We have updated our Servers since but the header size seems to keep
growing still. For reference, we have multiple bsd fileservers which are
used mostly over NFS, all with identical configuration (but varying
workload). They all still show these symptoms.
Any tips/hints/pointers are appreciated!
With kind regards,
Daniel
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
