Re: [zfs-discuss] sharenfs with over 10000 file systems
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Ahrens wrote: | I believe this is because sharemgr does an O(number of shares) operation | whenever you try to share/unshare anything (retrieving the list of shares | from the kernel to make sure that it isn't/is already shared). I couldn't | find a bug on this (though it's been known for some time), so feel free to | file a bug. Hope somebody is moving this to a hash or similar :-). - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ [EMAIL PROTECTED] http://www.argo.es/~jcea/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:[EMAIL PROTECTED] _/_/_/_/ _/_/_/_/_/ ~ _/_/ _/_/_/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/_/_/ _/_/_/_/ _/_/ "My name is Dump, Core Dump" _/_/_/_/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBR6IVMJlgi5GaxT1NAQLABwP+NA6Z9asuhdfLHq6E1g2pO4oCbYfYe+uG Aq29bIsdC8qOLyKyubAH4Mc+llF5BekZd8B/lGp2IPGXxJDjuvgXxHZ5W/SPk/2Y fTNPYaxnMO7JWxcIcATSl7zsQ2HkJTUbBXTuSRfxAtq52/g1vIc8W5kyBf4zvbCk RKPY6PBj0Vk= =oixV -END PGP SIGNATURE- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
Ok... So I was wrong. I was informed I had this backwards. It seems that this NFS4.1 mirror mounts thing is really only nice for getting rid of a lot of automount maps. You still have to share each filesystem :-( I hate it when I think there is hope just to have it taken away. Sigh... This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
On Mon, 28 Jan 2008, Chris wrote: > I did a little bit more digging and found some interesting things. NFS4 > Mirror mounts. This would seem to be the most logical option. In this > scenario the client would connect to a single mount /tank/users but would be > able to move through the individual user file systems underneath that mount. > i.e. /tank/users/bob or /tank/users/jenny. They are mounted automatically as > needed. Very cool. Now you share out the top of the tree while the nfs4 > server takes care of the rest. > > Read about it here: > > http://opensolaris.org/os/project/nfs-namespace/files/mm-PRS-open.html Actually this doesn't change the server side of things. You still need to share all the server file systems you want shared but the NFSv4 client needs to mount only the top-level file system (/tank/users in your case) and it automagically discovers the file systems underneath it. So, where you would have previously needed automounter maps to teach the client(s) about the server file systems you now have mirror mounts doing the same thing for you. Alok ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
I did a little bit more digging and found some interesting things. NFS4 Mirror mounts. This would seem to be the most logical option. In this scenario the client would connect to a single mount /tank/users but would be able to move through the individual user file systems underneath that mount. i.e. /tank/users/bob or /tank/users/jenny. They are mounted automatically as needed. Very cool. Now you share out the top of the tree while the nfs4 server takes care of the rest. Read about it here: http://opensolaris.org/os/project/nfs-namespace/files/mm-PRS-open.html watch this video http://frsun.downloads.edgesuite.net/sun/08A01141/ read more here http://blogs.sun.com/dtraub/entry/nfsv4_mirror_mounts_and_a This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
[EMAIL PROTECTED] wrote on 01/28/2008 09:11:53 AM: > I too am having the same issues. I started out using Solaris 10 > 8/07 release. I could create all the filesystems, 47,000 > filesystems, but if you needed to reboot, patch, shutdown Very > bad. So then I read about sharemgr and how it was supposed to > mitigate these issues. Well, after running a process that creates > zfs filesystems for all our users and then they get shared through > inherited zfs/nfs permissions I have only 6400 filesystems created > after 3 days of the process running. Not very good. I am going to > try and recreate the filesystems with sharenfs being done on each > filesystem rather than from the top of the tree to see if I can > improve the speed. In case you are wondering the system I have > configured is the following. > > Sun v245 with 16 gigs of RAM and 2 procs > 3 Apple X-Raids in a RAID-Z config -- each X-Raid is really like 2 > direct attached... so there are 6 independent connections to the > server. More like a RAID-Z set with 6 disks attached. > > You may ask why I would have that many filesystems. Well, as one of > the posters above pointed out, we had this same setup on an aging > sun 450 server. We had the 47k user home directories stored on it > with quotas enabled. Well, we needed to migrate this to a new > system and ZFS seemed like it was the answer to all our problems. > We have 16TB raw disk and needed a filesystem that could see this, > was robust, and very safe/secure. Enter zfs...maybe. Well, I can > see all the disk, I can create all the directories, I have done some > load testingImpressive. I just can't share them. > Chris, I am of firm belief that shoehorning zfs reservations/quotas to act as user/group quotas by creating filesystems is a very poor decision. I would a: wait until the ZFS team realize that they need to actually support real user/group quotas and implement them, or b: skip ZFS for this install -- the advantages of ZFS do not make up for the pain of losing real user/group quotas (or using fs reservations in lieu of real quotas). -Wade ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
I too am having the same issues. I started out using Solaris 10 8/07 release. I could create all the filesystems, 47,000 filesystems, but if you needed to reboot, patch, shutdown Very bad. So then I read about sharemgr and how it was supposed to mitigate these issues. Well, after running a process that creates zfs filesystems for all our users and then they get shared through inherited zfs/nfs permissions I have only 6400 filesystems created after 3 days of the process running. Not very good. I am going to try and recreate the filesystems with sharenfs being done on each filesystem rather than from the top of the tree to see if I can improve the speed. In case you are wondering the system I have configured is the following. Sun v245 with 16 gigs of RAM and 2 procs 3 Apple X-Raids in a RAID-Z config -- each X-Raid is really like 2 direct attached... so there are 6 independent connections to the server. More like a RAID-Z set with 6 disks attached. You may ask why I would have that many filesystems. Well, as one of the posters above pointed out, we had this same setup on an aging sun 450 server. We had the 47k user home directories stored on it with quotas enabled. Well, we needed to migrate this to a new system and ZFS seemed like it was the answer to all our problems. We have 16TB raw disk and needed a filesystem that could see this, was robust, and very safe/secure. Enter zfs...maybe. Well, I can see all the disk, I can create all the directories, I have done some load testingImpressive. I just can't share them. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
> New, yes. Aware - probably not. > > Given cheap filesystems, users would create "many" > filesystems was an easy guess, but I somehow don't > think anybody envisioned that users would be creating > tens of thousands of filesystems. > > ZFS - too good for it's own good :-p IMO (and given mails/posts I've seen typically by people using or wanting to use zfs at large universities and the like, for home directories) this is frequently driven by the need for per-user quotas. Since zfs doesn't have per-uid quotas, this means they end up creating (at least one) filesystem per user. That means a share per user, and locally a mount per user, which will never scale as well as (locally) a single share of /export/home, and a single mount (although there would of course be automounts to /home on demand, but they wouldn't slow down bootup). sharemgr and the like may be attempts to improve the situation, but they mitigate rather than eliminate the consequences of exploding what used to be a single large filesystem into a bunch of relatively small ones, simply based on the need to have per-user quotas with zfs. And there are still situations where a per-uid quota would be useful, such as /var/mail (although I could see that corrupting mailboxes in some cases) or other sorts of shared directories. OTOH, the implementation could certainly vary a little. The equivalent of the "quotas" file should be automatically created when quotas are enabled, and invisible; and unless quotas are not only disabled but purged somehow, it should maintain per-uid use statistics even for uids with no quotas, to eliminate the need for quotacheck (initialization of quotas might well be restricted to filesystem creation time, to eliminate the need for a cumbersome pass through existing data, at least at first; but that would probably be wanted too, since people don't always plan ahead). But other quota-related functionality could IMO maintain, although the implementations might have to get smarter, and there ought to be some alternative to the method presently used with ufs of simply reading the quotas file to iterate through the available stats. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
New, yes. Aware - probably not. Given cheap filesystems, users would create "many" filesystems was an easy guess, but I somehow don't think anybody envisioned that users would be creating tens of thousands of filesystems. ZFS - too good for it's own good :-p This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
I believe this is because sharemgr does an O(number of shares) operation whenever you try to share/unshare anything (retrieving the list of shares from the kernel to make sure that it isn't/is already shared). I couldn't find a bug on this (though it's been known for some time), so feel free to file a bug. --matt zfsmonk wrote: > Anyone out there using sharenfs=on with a large amount > of filesystems? We have over 1 filesystems all in one > pool. Everything is great until we turn on sharenfs > (zfs set sharenfs=on poolName). Once that is enabled, > zfs create poolName/filesystem takes about 5 minutes to > complete. If nfs sharing is turned off, completes right away. > > This happens on Solaris 807, b72 and b78. Has anyone seen this > behavior before? > > > This message posted from opensolaris.org > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
On Wed, Jan 23, 2008 at 08:02:22AM -0800, Akhilesh Mritunjai wrote: > I remember reading a discussion where these kind of problems were > discussed. > > Basically it boils down to "everything" not being aware of the > radical changes in "filesystems" concept. > > All these things are being worked on, but it might take sometime > before everything is made aware that yes it's no longer unusual that > there can be 1+ filesystems on one machine. But shouldn't sharemgr(1M) be "aware"? It's relatively new. -- albert chin ([EMAIL PROTECTED]) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] sharenfs with over 10000 file systems
I remember reading a discussion where these kind of problems were discussed. Basically it boils down to "everything" not being aware of the radical changes in "filesystems" concept. All these things are being worked on, but it might take sometime before everything is made aware that yes it's no longer unusual that there can be 1+ filesystems on one machine. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] sharenfs with over 10000 file systems
Anyone out there using sharenfs=on with a large amount of filesystems? We have over 1 filesystems all in one pool. Everything is great until we turn on sharenfs (zfs set sharenfs=on poolName). Once that is enabled, zfs create poolName/filesystem takes about 5 minutes to complete. If nfs sharing is turned off, completes right away. This happens on Solaris 807, b72 and b78. Has anyone seen this behavior before? This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
