Re: [Gluster-users] ZFS, ZIL and ARC
2016-10-30 15:05 GMT+01:00 Lindsay Mathieson : > zfs get secondarycache,primarycache # zfs get secondarycache,primarycache NAME PROPERTYVALUE SOURCE pci-:01:00.0-scsi-0:0:1:0secondarycache all default pci-:01:00.0-scsi-0:0:1:0primarycacheall default pci-:01:00.0-scsi-0:0:2:0secondarycache all default pci-:01:00.0-scsi-0:0:2:0primarycacheall default pci-:01:00.0-scsi-0:0:2:0@prova secondarycache all default pci-:01:00.0-scsi-0:0:2:0@prova primarycacheall default pci-:01:00.0-scsi-0:0:5:0secondarycache all default pci-:01:00.0-scsi-0:0:5:0primarycacheall default pci-:01:00.0-scsi-0:0:5:0@prova secondarycache all default pci-:01:00.0-scsi-0:0:5:0@prova primarycacheall default > > > Also > > zpool status # zpool status pool: pci-:01:00.0-scsi-0:0:1:0 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pci-:01:00.0-scsi-0:0:1:0ONLINE 0 0 0 pci-:01:00.0-scsi-0:0:1:0 ONLINE 0 0 0 errors: No known data errors pool: pci-:01:00.0-scsi-0:0:2:0 state: ONLINE scan: scrub repaired 0 in 1h9m with 0 errors on Sun Oct 30 15:16:43 2016 config: NAME STATE READ WRITE CKSUM pci-:01:00.0-scsi-0:0:2:0ONLINE 0 0 0 pci-:01:00.0-scsi-0:0:2:0 ONLINE 0 0 0 logs sdd1 ONLINE 0 0 0 cache sdd2 ONLINE 0 0 0 errors: No known data errors pool: pci-:01:00.0-scsi-0:0:5:0 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pci-:01:00.0-scsi-0:0:5:0ONLINE 0 0 0 pci-:01:00.0-scsi-0:0:5:0 ONLINE 0 0 0 errors: No known data errors > arc_summary.py -p 2 > > for stats on your ARC cache # ./arc_summary.py -p 2 ZFS Subsystem Report Sun Oct 30 18:59:59 2016 ARC Total accesses: 128.27m Cache Hit Ratio: 68.86% 88.33m Cache Miss Ratio: 31.14% 39.95m Actual Hit Ratio: 36.67% 47.04m Data Demand Efficiency: 98.67% 26.13m Data Prefetch Efficiency: 0.00% 21.68m CACHE HITS BY CACHE LIST: Anonymously Used: 43.25% 38.20m Most Recently Used: 31.54% 27.86m Most Frequently Used: 21.72% 19.18m Most Recently Used Ghost: 3.38% 2.99m Most Frequently Used Ghost: 0.10% 92.49k CACHE HITS BY DATA TYPE: Demand Data: 29.19% 25.79m Prefetch Data: 0.00% 9 Demand Metadata: 23.99% 21.19m Prefetch Metadata: 46.81% 41.35m CACHE MISSES BY DATA TYPE: Demand Data: 0.87% 348.07k Prefetch Data: 54.27% 21.68m Demand Metadata: 32.60% 13.02m Prefetch Metadata: 12.26% 4.90m > and > > arc_summary.py -p 3 > > for your l2arc cache # ./arc_summary.py -p 3 ZFS Subsystem Report Sun Oct 30 19:00:28 2016 L2 ARC Summary: (HEALTHY) Low Memory Aborts: 0 Free on Write: 2 R/W Clashes: 0 Bad Checksums: 0 IO Errors: 0 L2 ARC Size: (Adaptive) 3.58 GiB Compressed: 92.54% 3.31 GiB Header Size: 0.06% 2.25 MiB L2 ARC Evicts: Lock Retries: 0 Upon Reading: 0 L2 ARC Breakdown: 413.37k Hit Ratio: 0.00% 9 Miss Ratio: 100.00% 413.36k Feeds: 18.56k L2 ARC Writes: Writes Sent: 100.00% 2.31k ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ZFS, ZIL and ARC
2016-10-30 14:51 GMT+01:00 David Gossage : > Is it not showing as filling up or not showing any hit's to it? > > If you use `zpool iostat -v` does it show any data going to l2arc/cache or > any small usage on the zil/log? # zpool iostat -v capacity operationsbandwidth pool alloc free read write read write --- - - - - - - pci-:01:00.0-scsi-0:0:1:0 372K 1.81T 0 0 17 16 pci-:01:00.0-scsi-0:0:1:0 372K 1.81T 0 0 17 16 --- - - - - - - pci-:01:00.0-scsi-0:0:2:0 364G 1.46T138122 14.1M 14.1M pci-:01:00.0-scsi-0:0:2:0 364G 1.46T138122 14.1M 14.1M logs - - - - - - sdd1 0 7.94G 0 0 62223 cache- - - - - - sdd2 3.31G 100G 0 24 9 197K --- - - - - - - pci-:01:00.0-scsi-0:0:5:0 328G 1.49T 32 28 3.34M 3.36M pci-:01:00.0-scsi-0:0:5:0 328G 1.49T 32 28 3.34M 3.36M --- - - - - - - ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ZFS, ZIL and ARC
2016-10-30 14:49 GMT+01:00 David Gossage : > off top of my head I think zfs get all will get all settings for all > pools/vols etc.. Only pci-:01:00.0-scsi-0:0:2:0 is in use as brick, the other are only mounted but not part of any gluster cluster # zfs get all NAME PROPERTY VALUE SOURCE pci-:01:00.0-scsi-0:0:1:0type filesystem - pci-:01:00.0-scsi-0:0:1:0creation Sat Oct 29 13:08 2016 - pci-:01:00.0-scsi-0:0:1:0used 300K - pci-:01:00.0-scsi-0:0:1:0available 1.76T - pci-:01:00.0-scsi-0:0:1:0referenced96K - pci-:01:00.0-scsi-0:0:1:0compressratio 1.00x - pci-:01:00.0-scsi-0:0:1:0mounted yes - pci-:01:00.0-scsi-0:0:1:0quota none default pci-:01:00.0-scsi-0:0:1:0reservation none default pci-:01:00.0-scsi-0:0:1:0recordsize128K default pci-:01:00.0-scsi-0:0:1:0mountpoint /export/pci-_01_00.0-scsi-0_0_1_0 local pci-:01:00.0-scsi-0:0:1:0sharenfs off default pci-:01:00.0-scsi-0:0:1:0checksum on default pci-:01:00.0-scsi-0:0:1:0compression off default pci-:01:00.0-scsi-0:0:1:0atime on default pci-:01:00.0-scsi-0:0:1:0devices on default pci-:01:00.0-scsi-0:0:1:0exec on default pci-:01:00.0-scsi-0:0:1:0setuidon default pci-:01:00.0-scsi-0:0:1:0readonly off default pci-:01:00.0-scsi-0:0:1:0zoned off default pci-:01:00.0-scsi-0:0:1:0snapdir hidden default pci-:01:00.0-scsi-0:0:1:0aclinheritrestricted default pci-:01:00.0-scsi-0:0:1:0canmount on default pci-:01:00.0-scsi-0:0:1:0xattr on default pci-:01:00.0-scsi-0:0:1:0copies1 default pci-:01:00.0-scsi-0:0:1:0version 5 - pci-:01:00.0-scsi-0:0:1:0utf8only off - pci-:01:00.0-scsi-0:0:1:0normalization none - pci-:01:00.0-scsi-0:0:1:0casesensitivity sensitive - pci-:01:00.0-scsi-0:0:1:0vscan off default pci-:01:00.0-scsi-0:0:1:0nbmandoff default pci-:01:00.0-scsi-0:0:1:0sharesmb off default pci-:01:00.0-scsi-0:0:1:0refquota none default pci-:01:00.0-scsi-0:0:1:0refreservationnone default pci-:01:00.0-scsi-0:0:1:0primarycache all default pci-:01:00.0-scsi-0:0:1:0secondarycacheall default pci-:01:00.0-scsi-0:0:1:0usedbysnapshots 0 - pci-:01:00.0-scsi-0:0:1:0usedbydataset 96K - pci-:01:00.0-scsi-0:0:1:0usedbychildren204K - pci-:01:00.0-scsi-0:0:1:0usedbyrefreservation 0 - pci-:01:00.0-scsi-0:0:1:0logbias latency default pci-:01:00.0-scsi-0:0:1:0dedup off default pci-:01:00.0-scsi-0:0:1:0mlslabel none default pci-:01:00.0-scsi-0:0:1:0sync standard default pci-:01:00.0-scsi-0:0:1:0refcompressratio 1.00x - pci-:01:00.0-scsi-0:0:1:0written 96K - pci-:01:00.0-scsi-0:0:1:0logicalused 92.5K - pci-:01:00.0-scsi-0:0:1:0logicalreferenced 40K - pci-00
Re: [Gluster-users] ZFS, ZIL and ARC
On Sun, Oct 30, 2016 at 11:42:26PM +1000, Lindsay Mathieson wrote: > On 30/10/2016 11:38 PM, Gandalf Corvotempesta wrote: > > > > Currently i don't have any VM. I've just transferred my work maildir > > (about 80GB) to test a webmail and dovecot server through gluster mount > > > > Even by creating and reading huge test files i don't see any usage > > > > Whats your zfs primary and secondary cache set to? > > The zfs list might be more help here Whatever you find out on ZFS tuning and troubleshooting in relation to Gluster, it is much appreciated if you can provide additions and corrections to the docs. http://gluster.readthedocs.io/en/latest/Administrator%20Guide/Gluster%20On%20ZFS/ And this is the .md file that is used to generate the web page: https://github.com/gluster/glusterdocs/blob/master/Administrator%20Guide/Gluster%20On%20ZFS.md Thanks, Niels signature.asc Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ZFS, ZIL and ARC
On 30/10/2016 11:46 PM, Gandalf Corvotempesta wrote: This is my very first usage of zfs, please tell me the exact command to run because i don't know how to verify what you asked zfs get secondarycache,primarycache Also zpool status Will show us the layout of your pool. and: arc_summary.py -p 2 for stats on your ARC cache and arc_summary.py -p 3 for your l2arc cache -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ZFS, ZIL and ARC
On Sun, Oct 30, 2016 at 8:24 AM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > I've added an ssd to a zpool to be used as zil and l2arc cache. (One > partition for each service) > > Seems to be unused. I've tried with arcstat.sh script and i don't see any > increasing usage for the cache > > How can i debug the zil/arc cache usage? > Is it not showing as filling up or not showing any hit's to it? If you use `zpool iostat -v` does it show any data going to l2arc/cache or any small usage on the zil/log? > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ZFS, ZIL and ARC
off top of my head I think zfs get all will get all settings for all pools/vols etc.. *David Gossage* *Carousel Checks Inc. | System Administrator* *Office* 708.613.2284 On Sun, Oct 30, 2016 at 8:46 AM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > Il 30 ott 2016 14:42, "Lindsay Mathieson" > ha scritto: > > Whats your zfs primary and secondary cache set to? > > This is my very first usage of zfs, please tell me the exact command to > run because i don't know how to verify what you asked > > > The zfs list might be more help here > > I'm on mobile now, I'll reply when I'll be back at home > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ZFS, ZIL and ARC
Il 30 ott 2016 14:42, "Lindsay Mathieson" ha scritto: > Whats your zfs primary and secondary cache set to? This is my very first usage of zfs, please tell me the exact command to run because i don't know how to verify what you asked > The zfs list might be more help here I'm on mobile now, I'll reply when I'll be back at home ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ZFS, ZIL and ARC
On 30/10/2016 11:38 PM, Gandalf Corvotempesta wrote: Currently i don't have any VM. I've just transferred my work maildir (about 80GB) to test a webmail and dovecot server through gluster mount Even by creating and reading huge test files i don't see any usage Whats your zfs primary and secondary cache set to? The zfs list might be more help here -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ZFS, ZIL and ARC
Il 30 ott 2016 14:31, "Lindsay Mathieson" ha scritto: > > This is for VM Hosting? pretty normal. I've run l2arc for weeks with VM's on ZFS or gluster/zfs and only seen 4%-6% hit ratios. Between KVM caching and the zil, l2arc never gets a chance to shine. > Currently i don't have any VM. I've just transferred my work maildir (about 80GB) to test a webmail and dovecot server through gluster mount Even by creating and reading huge test files i don't see any usage ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ZFS, ZIL and ARC
On 30/10/2016 11:24 PM, Gandalf Corvotempesta wrote: I've added an ssd to a zpool to be used as zil and l2arc cache. (One partition for each service) Seems to be unused. I've tried with arcstat.sh script and i don't see any increasing usage for the cache How can i debug the zil/arc cache usage? This is for VM Hosting? pretty normal. I've run l2arc for weeks with VM's on ZFS or gluster/zfs and only seen 4%-6% hit ratios. Between KVM caching and the zil, l2arc never gets a chance to shine. -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
