Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-09 Thread Brandon Allbery 
On 10/9/13 08:01, "chas williams - CONTRACTOR" 
wrote:

>On Sat, 5 Oct 2013 20:13:22 +
>Coy Hile  wrote:
>
>> Along the same lines, is anybody using any of the Illumos distributions?
>
>I briefly ran an AFS fileserver on an Illumos x86 distribution
>(specifically OpenIndiana) inside a VM.  Unsurprisingly it worked.

For what it's worth, I tested http://wiki.openafs.org/SolarisQuickStart/
on OpenIndiana in a VM, and also ran through most of it on SmartOS
although it turned out SmartOS was not at all happy with the hardware I
had available (ancient cheap netbook, so not exactly surprising).

-- 
brandon s allbery kf8nhsine nomine associates
allber...@gmail.com   ballb...@sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-09 Thread chas williams - CONTRACTOR 
On Sat, 5 Oct 2013 20:13:22 +
Coy Hile  wrote:

> Along the same lines, is anybody using any of the Illumos distributions?

I briefly ran an AFS fileserver on an Illumos x86 distribution
(specifically OpenIndiana) inside a VM.  Unsurprisingly it worked.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-06 Thread Russ Allbery 
Ken Dreyer  writes:

> For example, DKMS would require me to install the compiler and various
> development libraries on every single AFS (or ZFS) system.

You can use DKMS to create installable packages and just install those
packages on the other systems.

> The third reason I find DKMS painful is the issue of repeatable builds.

Same solution there.  :)

-- 
Russ Allbery (r...@stanford.edu) 
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-06 Thread Ken Dreyer 
On Sun, Oct 6, 2013 at 12:36 AM, Dirk Heinrichs  wrote:
> Am Samstag 05 Oktober 2013, 18:16:54 schrieb Ken Dreyer:
>
>> The reason I have advocated against ZFS-on-Linux at work for our
>> fileservers is that out-of-tree modules on Linux are such a hassle.
>
> Hmm, not on Debian derivatives, thanks to DKMS.

Sure, I don't mean to imply that there aren't tools available to try
to make the process user-friendly. But I still find them painful.

For example, DKMS would require me to install the compiler and various
development libraries on every single AFS (or ZFS) system. That may be
acceptable for production systems, but it's not optimal for large
production server deployments. Moreover, it's not a good use of
resources to rebuild the same software over and over across all our
production systems. I'll grant that both of those are relatively minor
since disk space and CPU resources are getting cheaper, but at the
same time, the number of virtual machines in our environment is always
increasing.

The third reason I find DKMS painful is the issue of repeatable
builds. At work we use a binary-based OS distribution because we want
to reduce as much as possible the category of risk associated with
builds that can differ slightly from system to system. It's been my
experience that DKMS had the tendency to work or fail depending on
subtle configuration drift across systems. When considering production
servers, DKMS packages just don't have the same risk profile as a
regular binary package. From my experience, DKMS is one more area of
possible configuration entropy that I want to avoid.

I haven't use Debian enough to know how well DKMS works there. And on
Red Hat, I haven't used DKMS for a couple years. Please feel free to
take what I'm saying with a grain of salt :)

Of the the four options that I know:

1. Rebuild entirely new binaries for every new kernel version
2. DKMS
3. Use and trust Red Hat's kABI
4. Just avoid out-of-tree modules

each one has pros and cons, and none give me a warm fuzzy feeling,
particularly for something as critical as a file system.

- Ken
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-06 Thread Måns Nilsson 
Subject: RE: [OpenAFS] ZFS-on-Linux on production fileservers? Date: Sat, Oct 
05, 2013 at 08:13:22PM + Quoting Coy Hile (coy.h...@coyhile.com):
> Along the same lines, is anybody using any of the Illumos distributions?  
> Personally, I'm working on rolling my own SmartOS build that has the AFS 
> kernel module installed so that VMs could be AFS fileservers or clients. 
> (Correct me if I'm wrong, but the AFS kernel module is required only for the 
> clients and the fileservers, right? If I host is only an AFS dbserver, it has 
> no requirement for the kernel module to be loaded, does it?

You are quite correct. 

My server does not have the client code installed. I do not even build
the kernel module on it. I also run a second server machine on old 
Sparc iron, and neither it has the kernel module installed.

The only hassle is that some bos commands need  the
'-localauth' flag when run on the file server.

-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
Hello, GORRY-O!!  I'm a GENIUS from HARVARD!!


signature.asc
Description: Digital signature

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-05 Thread Dirk Heinrichs 
Am Samstag 05 Oktober 2013, 18:16:54 schrieb Ken Dreyer:

> The reason I have advocated against ZFS-on-Linux at work for our
> fileservers is that out-of-tree modules on Linux are such a hassle.

Hmm, not on Debian derivatives, thanks to DKMS.

Bye...

Dirk
-- 
Dirk Heinrichs 
Tel: +49 (0)2471 209385 | Mobil: +49 (0)176 34473913
GPG Public Key C2E467BB | Jabber: dirk.heinri...@altum.de


signature.asc
Description: This is a digitally signed message part.

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-05 Thread Ken Dreyer 
On Fri, Oct 4, 2013 at 8:31 AM, Jeff Blaine  wrote:
> * Are you using ZFS-on-Linux in production for file servers?
> * If not, and you looked into it, what stopped you?

The reason I have advocated against ZFS-on-Linux at work for our
fileservers is that out-of-tree modules on Linux are such a hassle.
It's painful enough to have to manage OpenAFS's kmod, and if we
started relying on ZFS for Linux, we'd be compounding that pain.

- Ken
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-05 Thread Jason Edgecombe 

On 10/05/2013 04:13 PM, Coy Hile wrote:

Along the same lines, is anybody using any of the Illumos distributions?  
Personally, I'm working on rolling my own SmartOS build that has the AFS kernel 
module installed so that VMs could be AFS fileservers or clients. (Correct me 
if I'm wrong, but the AFS kernel module is required only for the clients and 
the fileservers, right? If I host is only an AFS dbserver, it has no 
requirement for the kernel module to be loaded, does it?

-c


Only the client needs the kernel module. The file servers and db servers 
do not need the kernel module.


That said, you'll need an AFS client for running some admin commands, 
but that can be done from a separate machine from the 
fileserver/dbserver. For example, setting access rights on folders 
requires an AFS client.


Jason
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

RE: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-05 Thread Coy Hile 
Along the same lines, is anybody using any of the Illumos distributions?  
Personally, I'm working on rolling my own SmartOS build that has the AFS kernel 
module installed so that VMs could be AFS fileservers or clients. (Correct me 
if I'm wrong, but the AFS kernel module is required only for the clients and 
the fileservers, right? If I host is only an AFS dbserver, it has no 
requirement for the kernel module to be loaded, does it?

-c



From: openafs-info-ad...@openafs.org on behalf of Måns Nilsson
Sent: Saturday, October 5, 2013 6:55 PM
To: Jeff Blaine
Cc: OpenAFS
Subject: Re: [OpenAFS] ZFS-on-Linux on production fileservers?

Subject: [OpenAFS] ZFS-on-Linux on production fileservers? Date: Fri, Oct 04, 
2013 at 10:31:47AM -0400 Quoting Jeff Blaine (jbla...@kickflop.net):
> [ For those running ext3/ext4, a question further down for you as ]
> [ well!   ]
>
> We're still a 100% Solaris + ZFS file server shop. We're EOLing
> our Sun SPARC hardware (with tears in our eyes) this year.
>
> Before we spend a significant amount of time evaluating this, I
> figured I'd ask first. Any brief response would be greatly appre-
> ciated. The generously longer the better :)
>
> * Are you using ZFS-on-Linux in production for file servers?
> * If not, and you looked into it, what stopped you?
> * If you are, how is it working out for you?
>
> ext3/ext4 people: What is your fsck strategy?

I'd second the question about FreeBSD that was asked earlier. My
personal cell runs vicep* on ZFS, 4 zmirrored 2TB SATA drives on an old
Dell 2950 with 8G RAM. I also iSCSI share a few devices (around a TB,
mail spool, backup spool, and scratch partition) from it. I'm very
happy. Rock solid; as good as Solaris on X86 but Larry does not even
get credit. Performance-wise, perhaps not super hot, but enough.

The only important part is that you MUST NOT have a 32-bit kernel. Now,
today, that probably is the default.

--
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
I'm RELIGIOUS!!  I love a man with a HAIRPIECE!!  Equip me with MISSILES!!
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-05 Thread Måns Nilsson 
Subject: [OpenAFS] ZFS-on-Linux on production fileservers? Date: Fri, Oct 04, 
2013 at 10:31:47AM -0400 Quoting Jeff Blaine (jbla...@kickflop.net):
> [ For those running ext3/ext4, a question further down for you as ]
> [ well!   ]
> 
> We're still a 100% Solaris + ZFS file server shop. We're EOLing
> our Sun SPARC hardware (with tears in our eyes) this year.
> 
> Before we spend a significant amount of time evaluating this, I
> figured I'd ask first. Any brief response would be greatly appre-
> ciated. The generously longer the better :)
> 
> * Are you using ZFS-on-Linux in production for file servers?
> * If not, and you looked into it, what stopped you?
> * If you are, how is it working out for you?
> 
> ext3/ext4 people: What is your fsck strategy?

I'd second the question about FreeBSD that was asked earlier. My
personal cell runs vicep* on ZFS, 4 zmirrored 2TB SATA drives on an old
Dell 2950 with 8G RAM. I also iSCSI share a few devices (around a TB,
mail spool, backup spool, and scratch partition) from it. I'm very
happy. Rock solid; as good as Solaris on X86 but Larry does not even
get credit. Performance-wise, perhaps not super hot, but enough.

The only important part is that you MUST NOT have a 32-bit kernel. Now,
today, that probably is the default.

-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
I'm RELIGIOUS!!  I love a man with a HAIRPIECE!!  Equip me with MISSILES!!


signature.asc
Description: Digital signature

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread Harald Barth 

> * Are you using ZFS-on-Linux in production for file servers?

Yes.

> * If not, and you looked into it, what stopped you?

Long there was fear and doubt, but the (not) "quality" of HW-Raid
solutions and hassle of Linux SW-Raid convinced us that it could not
be worse with ZFS.

> * If you are, how is it working out for you?

It does.

The are-the-zpools-OK reporting could be more comfortable and
the zpool status output is not compatible to the old one.
But compared to problems we had before, that's a no-brainer.

Don't be surprised if raidz needs CPU power to calculate the
checksums, so you need to get the balance between I/O and CPU/cores
right for the raidz level you want.

> ext3/ext4 people: What is your fsck strategy?

Before that we used xfs on HW- and SW-Raid. We had no problems with
the xfs part of it. However we felt all the time that the possible max
log sizes were 1990-ish.

Harald.
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread Markus Koeberl 
On Friday 04 October 2013 16:31:47 Jeff Blaine wrote:
> [ For those running ext3/ext4, a question further down for you as ]
> [ well!   ]
> 
> We're still a 100% Solaris + ZFS file server shop. We're EOLing
> our Sun SPARC hardware (with tears in our eyes) this year.
> 
> Before we spend a significant amount of time evaluating this, I
> figured I'd ask first. Any brief response would be greatly appre-
> ciated. The generously longer the better :)
> 
> * Are you using ZFS-on-Linux in production for file servers?

At home I started playing about two years ago.
I am now using a 3 disk raidz and a mirror. Had big problems with my hardware 
at the beginning. But thanks to ZFS I was able to eliminate the bad disks. Now 
everything is working.

At work I have one server running for 2 months now without big troubles
~30 linux $HOME volumes on a SSD mirror
~350 RO and 250 RW volumes on 12x2TB sata mirror +l2arc

> * If not, and you looked into it, what stopped you?
> * If you are, how is it working out for you?

I set arc_max to 8GB on a 24GB machine which lets me 1.9GB free at the moment. 
There have been suggestions on the mailinglist to play with arc_metadata_max 
and to set vm.min_free_kbytes to about 1% of RAM to avoid huge load or a lock 
up. I have not played around with this because limiting arc to 8GB es good 
enough for me at the moment. With compression set to lz4 it gives me good 
performance.


I strongly suggest to test well and to have a look at the current limitations 
of zfsonlinux (there have been several discussions on the mailing list). 


regards
Markus
-- 
Markus Koeberl
Graz University of Technology
Signal Processing and Speech Communication Laboratory
E-mail: markus.koeb...@tugraz.at
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread James E. Dobson 


FWIW: I'm using SmartOS (USB device boot) to boot a "smart machine" (aka 
a "zone") with OpenAFS. Giving up any large amount of space for boot 
devices on my fileservers seems rather stupid. This has been in 
production for several months now.


Using Debian + zfsonlinux with some success for other systems.

Jed Dobson

Dartmouth College
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread Dirk Heinrichs 
Am Freitag 04 Oktober 2013, 17:18:24 schrieb mi...@task.gda.pl:

> http://conferences.inf.ed.ac.uk/eakc2012/slides/AFS_on_Solaris_ZFS.pdf

Thanks a lot.

Bye...

Dirk
-- 
Dirk Heinrichs 
Tel: +49 (0)2471 209385 | Mobil: +49 (0)176 34473913
GPG Public Key C2E467BB | Jabber: dirk.heinri...@altum.de


signature.asc
Description: This is a digitally signed message part.

RE: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread milek 


> -Original Message-
> From: openafs-info-ad...@openafs.org [mailto:openafs-info-
> ad...@openafs.org] On Behalf Of Dirk Heinrichs
> Sent: 04 October 2013 17:08
> To: openafs-info@openafs.org
> Subject: Re: [OpenAFS] ZFS-on-Linux on production fileservers?
> 
> Am Freitag 04 Oktober 2013, 16:51:28 schrieb mi...@task.gda.pl:
> 
> > See my presentation about it last year.
> 
> Link?


http://conferences.inf.ed.ac.uk/eakc2012/slides/AFS_on_Solaris_ZFS.pdf

-- 
Robert Milkowski
http://milek.blogspot.com


___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread Stephan Wiesand 

On Oct 4, 2013, at 18:08 , Dirk Heinrichs wrote:

> Am Freitag 04 Oktober 2013, 16:51:28 schrieb mi...@task.gda.pl:
> 
>> See my presentation about it last year.
> 
> Link?

http://conferences.inf.ed.ac.uk/eakc2012/

-- 
Stephan Wiesand
DESY -DV-
Platanenenallee 6
15738 Zeuthen, Germany

___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread Dirk Heinrichs 
Am Freitag 04 Oktober 2013, 16:51:28 schrieb mi...@task.gda.pl:

> See my presentation about it last year.

Link?

Bye...

Dirk
-- 
Dirk Heinrichs 
Tel: +49 (0)2471 209385 | Mobil: +49 (0)176 34473913
GPG Public Key C2E467BB | Jabber: dirk.heinri...@altum.de


signature.asc
Description: This is a digitally signed message part.

RE: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread milek 


> -Original Message-
> From: openafs-info-ad...@openafs.org [mailto:openafs-info-
> ad...@openafs.org] On Behalf Of Jeff Blaine
> Sent: 04 October 2013 15:32
> To: OpenAFS
> Subject: [OpenAFS] ZFS-on-Linux on production fileservers?
> 
> [ For those running ext3/ext4, a question further down for you as ]
> [ well!   ]
> 
> We're still a 100% Solaris + ZFS file server shop. We're EOLing our Sun
> SPARC hardware (with tears in our eyes) this year.
> 

Why not ZFS on Solaris 11 x86? You can run it on non-Oracle hardware if you
prefer.
We have a pretty large installation on top of Solaris 11 x86 + ZFS, both on
Oracle and 3rd party x86 servers.
See my presentation about it last year.


--
Robert Milkowski
http://milek.blogspot.com



___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread Dirk Heinrichs 
Am Freitag 04 Oktober 2013, 10:31:47 schrieb Jeff Blaine:

> We're still a 100% Solaris + ZFS file server shop. We're EOLing
> our Sun SPARC hardware (with tears in our eyes) this year.
> 
> Before we spend a significant amount of time evaluating this, I
> figured I'd ask first. Any brief response would be greatly appre-
> ciated. The generously longer the better :)
> 
> * Are you using ZFS-on-Linux in production for file servers?
> * If not, and you looked into it, what stopped you?
> * If you are, how is it working out for you?

A couple of weeks ago, I tried to install a _desktop_ system on ZFSonLinux. 
Can't remember the exact reason, but I quickly decided to stick with a native 
Linux FS.

OTOH, I run my own small home cell on an Arm box (Guruplug) using btrfs (both 
vicepXX and client cache). If it must be ZFS, would FreeBSD be an option?

Bye...

Dirk
-- 
Dirk Heinrichs 
Tel: +49 (0)2471 209385 | Mobil: +49 (0)176 34473913
GPG Public Key C2E467BB | Jabber: dirk.heinri...@altum.de


signature.asc
Description: This is a digitally signed message part.

Re: [OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread Dan Van Der Ster 

On Oct 4, 2013, at 4:31 PM, Jeff Blaine 
 wrote:

> * If not, and you looked into it, what stopped you?

We stopped because of the memory fragmentation issue. ZFS will use ~twice the 
arc limit you set, and (in my experience) if you don't set the it wisely (e.g. 
20-25% total memory), your server will lock up solid.

Cheers, Dan
CERN IT___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

[OpenAFS] ZFS-on-Linux on production fileservers?

2013-10-04 Thread Jeff Blaine 
[ For those running ext3/ext4, a question further down for you as ]
[ well!   ]

We're still a 100% Solaris + ZFS file server shop. We're EOLing
our Sun SPARC hardware (with tears in our eyes) this year.

Before we spend a significant amount of time evaluating this, I
figured I'd ask first. Any brief response would be greatly appre-
ciated. The generously longer the better :)

* Are you using ZFS-on-Linux in production for file servers?
* If not, and you looked into it, what stopped you?
* If you are, how is it working out for you?

ext3/ext4 people: What is your fsck strategy?
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info