RE: [OpenAFS] Re: [ Openafs : cache on zfs ]
> -Original Message- > From: openafs-info-ad...@openafs.org [mailto:openafs-info- > ad...@openafs.org] On Behalf Of Andrew Deason > Sent: 03 October 2013 19:17 > To: openafs-info@openafs.org > Subject: [OpenAFS] Re: [ Openafs : cache on zfs ] > > On Thu, 3 Oct 2013 19:34:27 +0200 > nicolas prochazka wrote: > > > Hello again , > > after some tests to use zfs as afs cache, linux kernel tells : > > BUG : soft lockup - CPU0 stuck for 23s ! [ afs_cachetrim:2908] > > > > Any ideas are welcome, > > It seems pretty likely from your other message that using zfsonlinux > for the openafs client cache is not going to work at all until someone > takes the time to add support for it. Just use another filesystem. > > Even on other platforms, ZFS has some characteristics that make it not > ideal for a cache, and in the past I've recommended using something > else when it's easy to do so (e.g. UFS on Solaris, even if it's jus on > a zvol). At least on Solaris, ZFS does some somewhat unique things with > space allocations that have created some semi-unavoidable problems for > the cache manager. > > I can understand if someone is using ZFS for their root fs, and they > don't want to make a separate fs just for the cache, but if you're > making an fs just for the cache or something, it doesn't make a lot of > sense. We've been running AFS cache on ZFS directly for some time now, with no issues. The trick is to set AFS cache size somewhat smaller than the amount of space present by the file system (so essentially we slightly over-provision). On top of it we enable ZFS compression on the cache file system, so in reality we overprovision even more. All works fine. Given that AFS cache tends to be small (several GBs, anyone is using considerably larger cache?), a small over-provisioning is not an issue. For example, create ZFS file system for AFS cache as: # zfs create -o atime=off -o quota=8G -o reservation=8G -o compression=lzjb -o recordsize=8K -o mountpoint=/afscache rpool/afscache And in cacheinfo the specified cache size is: 6710886 This is less than 2GB overprovisioned (assuming nothing will compress), but with modern disk sizes of OS it doesn't matter at all and seems to be a safe enough margin. The compression is not for disk space saving it is more for doing less i/o in some cases (depending on data). -- Robert Milkowski http://milek.blogspot.com ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
RE: [OpenAFS] Re: [ Openafs : cache on zfs ]
> -Original Message- > From: openafs-info-ad...@openafs.org [mailto:openafs-info- > ad...@openafs.org] On Behalf Of Andrew Deason > Sent: 03 October 2013 19:17 > To: openafs-info@openafs.org > Subject: [OpenAFS] Re: [ Openafs : cache on zfs ] > > On Thu, 3 Oct 2013 19:34:27 +0200 > nicolas prochazka wrote: > > > Hello again , > > after some tests to use zfs as afs cache, linux kernel tells : > > BUG : soft lockup - CPU0 stuck for 23s ! [ afs_cachetrim:2908] > > > > Any ideas are welcome, > > It seems pretty likely from your other message that using zfsonlinux > for the openafs client cache is not going to work at all until someone > takes the time to add support for it. Just use another filesystem. > > Even on other platforms, ZFS has some characteristics that make it not > ideal for a cache, and in the past I've recommended using something > else when it's easy to do so (e.g. UFS on Solaris, even if it's jus on > a zvol). At least on Solaris, ZFS does some somewhat unique things with > space allocations that have created some semi-unavoidable problems for > the cache manager. > > I can understand if someone is using ZFS for their root fs, and they > don't want to make a separate fs just for the cache, but if you're > making an fs just for the cache or something, it doesn't make a lot of > sense. We've been running AFS cache on ZFS directly for some time now, with no issues. The trick is to set AFS cache size somewhat smaller than the amount of space present by the file system (so essentially we slightly over-provision). On top of it we enable ZFS compression on the cache file system, so in reality we overprovision even more. All works fine. Given that AFS cache tends to be small (several GBs, anyone is using considerably larger cache?), a small over-provisioning is not an issue. For example, create ZFS file system for AFS cache as: # zfs create -o atime=off -o quota=8G -o reservation=8G -o compression=lzjb -o recordsize=8K -o mountpoint=/afscache rpool/afscache And in cacheinfo the specified cache size is: 6710886 This is less than 2GB overprovisioned (assuming nothing will compress), but with modern disk sizes of OS it doesn't matter at all and seems to be a safe enough margin. The compression is not for disk space saving it is more for doing less i/o in some cases (depending on data). -- Robert Milkowski http://milek.blogspot.com ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: [ Openafs : cache on zfs ]
On 10/03/2013 02:16 PM, Andrew Deason wrote: On Thu, 3 Oct 2013 19:34:27 +0200 nicolas prochazka wrote: Hello again , after some tests to use zfs as afs cache, linux kernel tells : BUG : soft lockup - CPU0 stuck for 23s ! [ afs_cachetrim:2908] Any ideas are welcome, It seems pretty likely from your other message that using zfsonlinux for the openafs client cache is not going to work at all until someone takes the time to add support for it. Just use another filesystem. Even on other platforms, ZFS has some characteristics that make it not ideal for a cache, and in the past I've recommended using something else when it's easy to do so (e.g. UFS on Solaris, even if it's jus on a zvol). At least on Solaris, ZFS does some somewhat unique things with space allocations that have created some semi-unavoidable problems for the cache manager. I can understand if someone is using ZFS for their root fs, and they don't want to make a separate fs just for the cache, but if you're making an fs just for the cache or something, it doesn't make a lot of sense. My impression has always been that journalling is not only not needed for the afs cache, but even not desirable. Up until recently I'd use the simplest thing - an ext2 partition with noatime. On my latest installs I've begun using ext4 with the journal turned off, figuring that with any luck afs cache blocks would map to ext4 extents, giving me better cache performance than even ext2. Then again somewhere else, on top of no journal, I got the options: /dev/sdb6/var/cache/openafsext4 data=writeback,barrier=0,nobh,errors=remount-ro,noatime1 2 Dale Pontius -- Dale Pontius Senior Engineer IBM Corporation Phone: (802) 769-6850 Tie-Line: 446-6850 email: pont...@us.ibm.com This e-mail and its attachments, if any, may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply e-mail and delete all copies of this message from your system without copying it and notify sender of the misdirection by reply e-mail. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Re: [ Openafs : cache on zfs ]
On 10/3/13 14:08, "nicolas prochazka" wrote: >hello, >sorry for the spam, this is a misconfigured cache option. >Regards, >Nicolas Prochazka. > >2013/10/3 nicolas prochazka : >> Hello again , >> after some tests to use zfs as afs cache, >> linux kernel tells : >> BUG : soft lockup - CPU0 stuck for 23s ! [ afs_cachetrim:2908] Someone should probably mention that OpenAFS cache on ZFS has performance issues (great care is needed to get cache chunks to mesh well with ZFS blocks and stripes; the latter are not tunable) and it's generally easier and preferable to use a fast filesystem on a zvolume. -- brandon s allbery kf8nhsine nomine associates allber...@gmail.com ballb...@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad
