RE: [OpenAFS] Re: [ Openafs : cache on zfs ]

2013-10-04 Thread milek


> -Original Message-
> From: openafs-info-ad...@openafs.org [mailto:openafs-info-
> ad...@openafs.org] On Behalf Of Andrew Deason
> Sent: 03 October 2013 19:17
> To: openafs-info@openafs.org
> Subject: [OpenAFS] Re: [ Openafs : cache on zfs ]
> 
> On Thu, 3 Oct 2013 19:34:27 +0200
> nicolas prochazka  wrote:
> 
> > Hello again ,
> > after some tests to use zfs as afs cache, linux kernel tells :
> > BUG : soft lockup - CPU0 stuck for 23s ! [ afs_cachetrim:2908]
> >
> > Any ideas are welcome,
> 
> It seems pretty likely from your other message that using zfsonlinux
> for the openafs client cache is not going to work at all until someone
> takes the time to add support for it. Just use another filesystem.
> 
> Even on other platforms, ZFS has some characteristics that make it not
> ideal for a cache, and in the past I've recommended using something
> else when it's easy to do so (e.g. UFS on Solaris, even if it's jus on
> a zvol). At least on Solaris, ZFS does some somewhat unique things with
> space allocations that have created some semi-unavoidable problems for
> the cache manager.
> 
> I can understand if someone is using ZFS for their root fs, and they
> don't want to make a separate fs just for the cache, but if you're
> making an fs just for the cache or something, it doesn't make a lot of
> sense.


We've been running AFS cache on ZFS directly for some time now, with no
issues.
The trick is to set AFS cache size somewhat smaller than the amount of space
present by the file system (so essentially we slightly over-provision). On
top of it we enable ZFS compression on the cache file system, so in reality
we overprovision even more. All works fine. 

Given that AFS cache tends to be small (several GBs, anyone is using
considerably larger cache?), a small over-provisioning is not an issue.

For example, create ZFS file system for AFS cache as:

# zfs create -o atime=off -o quota=8G -o reservation=8G -o compression=lzjb
-o recordsize=8K -o mountpoint=/afscache rpool/afscache

And in cacheinfo the specified cache size is: 6710886

This is less than 2GB overprovisioned (assuming nothing will compress), but
with modern disk sizes of OS it doesn't matter at all and seems to be a safe
enough margin. The compression is not for disk space saving it is more for
doing less i/o in some cases (depending on data).


--
Robert Milkowski
http://milek.blogspot.com


___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info


RE: [OpenAFS] Re: [ Openafs : cache on zfs ]

2013-10-04 Thread milek


> -Original Message-
> From: openafs-info-ad...@openafs.org [mailto:openafs-info-
> ad...@openafs.org] On Behalf Of Andrew Deason
> Sent: 03 October 2013 19:17
> To: openafs-info@openafs.org
> Subject: [OpenAFS] Re: [ Openafs : cache on zfs ]
> 
> On Thu, 3 Oct 2013 19:34:27 +0200
> nicolas prochazka  wrote:
> 
> > Hello again ,
> > after some tests to use zfs as afs cache, linux kernel tells :
> > BUG : soft lockup - CPU0 stuck for 23s ! [ afs_cachetrim:2908]
> >
> > Any ideas are welcome,
> 
> It seems pretty likely from your other message that using zfsonlinux
> for the openafs client cache is not going to work at all until someone
> takes the time to add support for it. Just use another filesystem.
> 
> Even on other platforms, ZFS has some characteristics that make it not
> ideal for a cache, and in the past I've recommended using something
> else when it's easy to do so (e.g. UFS on Solaris, even if it's jus on
> a zvol). At least on Solaris, ZFS does some somewhat unique things with
> space allocations that have created some semi-unavoidable problems for
> the cache manager.
> 
> I can understand if someone is using ZFS for their root fs, and they
> don't want to make a separate fs just for the cache, but if you're
> making an fs just for the cache or something, it doesn't make a lot of
> sense.

We've been running AFS cache on ZFS directly for some time now, with no
issues.
The trick is to set AFS cache size somewhat smaller than the amount of space
present by the file system (so essentially we slightly over-provision). On
top of it we enable ZFS compression on the cache file system, so in reality
we overprovision even more. All works fine. 

Given that AFS cache tends to be small (several GBs, anyone is using
considerably larger cache?), a small over-provisioning is not an issue.

For example, create ZFS file system for AFS cache as:

# zfs create -o atime=off -o quota=8G -o reservation=8G -o compression=lzjb
-o recordsize=8K -o mountpoint=/afscache rpool/afscache

And in cacheinfo the specified cache size is: 6710886

This is less than 2GB overprovisioned (assuming nothing will compress), but
with modern disk sizes of OS it doesn't matter at all and seems to be a safe
enough margin. The compression is not for disk space saving it is more for
doing less i/o in some cases (depending on data).


--
Robert Milkowski
http://milek.blogspot.com


___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info


Re: [OpenAFS] Re: [ Openafs : cache on zfs ]

2013-10-03 Thread Dale Pontius

On 10/03/2013 02:16 PM, Andrew Deason wrote:

On Thu, 3 Oct 2013 19:34:27 +0200
nicolas prochazka  wrote:


Hello again ,
after some tests to use zfs as afs cache,
linux kernel tells :
BUG : soft lockup - CPU0 stuck for 23s ! [ afs_cachetrim:2908]

Any ideas are welcome,

It seems pretty likely from your other message that using zfsonlinux for
the openafs client cache is not going to work at all until someone takes
the time to add support for it. Just use another filesystem.

Even on other platforms, ZFS has some characteristics that make it not
ideal for a cache, and in the past I've recommended using something else
when it's easy to do so (e.g. UFS on Solaris, even if it's jus on a
zvol). At least on Solaris, ZFS does some somewhat unique things with
space allocations that have created some semi-unavoidable problems for
the cache manager.

I can understand if someone is using ZFS for their root fs, and they
don't want to make a separate fs just for the cache, but if you're
making an fs just for the cache or something, it doesn't make a lot of
sense.

My impression has always been that journalling is not only not needed 
for the afs cache, but even not desirable.  Up until recently I'd use 
the simplest thing - an ext2 partition with noatime.  On my latest 
installs I've begun using ext4 with the journal turned off, figuring 
that with any luck afs cache blocks would map to ext4 extents, giving me 
better cache performance than even ext2.  Then again somewhere else, on 
top of no journal, I got the options:


/dev/sdb6/var/cache/openafsext4 
data=writeback,barrier=0,nobh,errors=remount-ro,noatime1 2


Dale Pontius

--
Dale Pontius
Senior Engineer
IBM Corporation
Phone: (802) 769-6850
Tie-Line: 446-6850
email: pont...@us.ibm.com

This e-mail and its attachments, if any, may contain confidential and 
privileged material for the sole use of the intended recipient. Any review, 
use, distribution or disclosure by others is strictly prohibited. If you are 
not the intended recipient (or authorized to receive for the recipient), please 
contact the sender by reply e-mail and delete all copies of this message from 
your system without copying it and notify sender of the misdirection by reply 
e-mail.

___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info


Re: [OpenAFS] Re: [ Openafs : cache on zfs ]

2013-10-03 Thread Brandon Allbery
On 10/3/13 14:08, "nicolas prochazka"  wrote:

>hello,
>sorry for the spam, this is a misconfigured cache option.
>Regards,
>Nicolas Prochazka.
>
>2013/10/3 nicolas prochazka :
>> Hello again ,
>> after some tests to use zfs as afs cache,
>> linux kernel tells :
>> BUG : soft lockup - CPU0 stuck for 23s ! [ afs_cachetrim:2908]

Someone should probably mention that OpenAFS cache on ZFS has performance
issues (great care is needed to get cache chunks to mesh well with ZFS
blocks and stripes; the latter are not tunable) and it's generally easier
and preferable to use a fast filesystem on a zvolume.

-- 
brandon s allbery kf8nhsine nomine associates
allber...@gmail.com   ballb...@sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad