On 10 November, 2006 - Sanjeev Bagewadi sent me these 3,5K bytes:
> Comments in line...
>
> Neil Perrin wrote:
>
> >>>>1. DNLC-through-ZFS doesn't seem to listen to ncsize.
> >>>>
> >>>>The filesystem currently has ~550k inodes and large portions of it is
> >>>>frequently looked over with rsync (over nfs). mdb said ncsize was
> >>>>about
> >>>>68k and vmstat -s said we had a hitrate of ~30%, so I set ncsize to
> >>>>600k and rebooted.. Didn't seem to change much, still seeing
> >>>>hitrates at
> >>>>about the same and manual find(1) doesn't seem to be that cached
> >>>>(according to vmstat and dnlcsnoop.d).
> >>>>When booting, the following message came up, not sure if it matters
> >>>>or not:
> >>>>NOTICE: setting nrnode to max value of 351642
> >>>>NOTICE: setting nrnode to max value of 235577
> >>>>
> >>>>Is there a separate ZFS-DNLC knob to adjust for this? Wild guess is
> >>>>that
> >>>>it has its own implementation which is integrated with the rest of the
> >>>>ZFS cache which throws out metadata cache in favour of data cache.. or
> >>>>something..
> >>>
> >>Current memory usage (for some values of usage ;):
> >># echo ::memstat|mdb -k
> >>Page Summary Pages MB %Tot
> >>------------ ---------------- ---------------- ----
> >>Kernel 95584 746 75%
> >>Anon 20868 163 16%
> >>Exec and libs 1703 13 1%
> >>Page cache 1007 7 1%
> >>Free (cachelist) 97 0 0%
> >>Free (freelist) 7745 60 6%
> >>
> >>Total 127004 992
> >>Physical 125192 978
> >>
> >>
> >>/Tomas
> >>
> >>
> >This memory usage shows nearly all of memory consumed by the kernel
> >and probably by ZFS. ZFS can't add any more DNLC entries due to lack of
> >memory without purging others. This can be seen from the number of
> >dnlc_nentries being way less than ncsize.
> >I don't know if there's a DMU or ARC bug to reduce the memory footprint
> >of their internal structures for situations like this, but we are
> >aware of the
> >issue.
>
> Can you please check the zio buffers and the arc status ?
>
> Here is how you can do it :
> - Start mdb : ie. mdb -k
>
> > ::kmem_cache
>
> - In the output generated above check the amount consumed by the
> zio_buf_*, arc_buf_t and
> arc_buf_hdr_t.
ADDR NAME FLAG CFLAG BUFSIZE BUFTOTL
0000030002640a08 zio_buf_512 0000 020000 512 102675
0000030002640c88 zio_buf_1024 0200 020000 1024 48
0000030002640f08 zio_buf_1536 0200 020000 1536 70
0000030002641188 zio_buf_2048 0200 020000 2048 16
0000030002641408 zio_buf_2560 0200 020000 2560 9
0000030002641688 zio_buf_3072 0200 020000 3072 16
0000030002641908 zio_buf_3584 0200 020000 3584 18
0000030002641b88 zio_buf_4096 0200 020000 4096 12
0000030002668008 zio_buf_5120 0200 020000 5120 32
0000030002668288 zio_buf_6144 0200 020000 6144 8
0000030002668508 zio_buf_7168 0200 020000 7168 1032
0000030002668788 zio_buf_8192 0200 020000 8192 8
0000030002668a08 zio_buf_10240 0200 020000 10240 8
0000030002668c88 zio_buf_12288 0200 020000 12288 4
0000030002668f08 zio_buf_14336 0200 020000 14336 468
0000030002669188 zio_buf_16384 0200 020000 16384 3326
0000030002669408 zio_buf_20480 0200 020000 20480 16
0000030002669688 zio_buf_24576 0200 020000 24576 3
0000030002669908 zio_buf_28672 0200 020000 28672 12
0000030002669b88 zio_buf_32768 0200 020000 32768 1935
000003000266c008 zio_buf_40960 0200 020000 40960 13
000003000266c288 zio_buf_49152 0200 020000 49152 9
000003000266c508 zio_buf_57344 0200 020000 57344 7
000003000266c788 zio_buf_65536 0200 020000 65536 3272
000003000266ca08 zio_buf_73728 0200 020000 73728 10
000003000266cc88 zio_buf_81920 0200 020000 81920 7
000003000266cf08 zio_buf_90112 0200 020000 90112 5
000003000266d188 zio_buf_98304 0200 020000 98304 7
000003000266d408 zio_buf_106496 0200 020000 106496 12
000003000266d688 zio_buf_114688 0200 020000 114688 6
000003000266d908 zio_buf_122880 0200 020000 122880 5
000003000266db88 zio_buf_131072 0200 020000 131072 92
0000030002670508 arc_buf_hdr_t 0000 000000 128 11970
0000030002670788 arc_buf_t 0000 000000 40 7308
> - Dump the values of arc
>
> > arc::print struct arc
> arc::print struct arc
{
anon = ARC_anon
mru = ARC_mru
mru_ghost = ARC_mru_ghost
mfu = ARC_mfu
mfu_ghost = ARC_mfu_ghost
size = 0x6f7a400
p = 0x5d9bd5a
c = 0x5f6375a
c_min = 0x4000000
c_max = 0x2e82a000
hits = 0x40e0a15
misses = 0x1cec4a4
deleted = 0x1b0ba0d
skipped = 0x24ea64e13
hash_elements = 0x179d
hash_elements_max = 0x60bb
hash_collisions = 0x8dca3a
hash_chains = 0x391
hash_chain_max = 0x8
no_grow = 0x1
}
So, about 100MB and a memory crunch..
> size = 0x3e20000 <-- tells you the current memory
> consumed by ARC buffer (including the
> the memory consumed
> for the data cached ie. zio_buff_*
> no_grow = 0 <-- This would be set to 1 if we have a
> memory crunch
> And as Niel pointed out we would probably need some way of limiting the
> ARC consumption.
/Tomas
--
Tomas Ögren, [EMAIL PROTECTED], http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
