Re: [zfs-discuss] [illumos-Developer] revisiting aclmode options
On 8/2/2011 7:07 AM, Gordon Ross wrote: It seems consistent to me that a "discard" mode would simply never present suid/sgid/sticky. (It discards mode settings.) After all, the suid/sgid/sticky bits don't have any counterpart in Windows security descriptors, and Windows ACL use interited $CREATOR_OWNER ACEs to do the equivalent of the sticky bit. I see it somewhat differently; the purpose of "discard" is to prevent any attempted change of the mode bits via chmod from affecting the ACL. As you point out, there is no corresponding functionality in NFSv4 ACLs, so by definition a change of the suid/sgid/sticky part of the mode bits would not affect the ACL. And not allowing them to be changed would result in lost functionality -- for example, setting the sgid bit on the directory so the group owner is inherited on child directories, which is actually quite valuable for the functionality of the group@ entry. So I think the implementation of both a "discard" and "deny" aclmode would need to incorporate the ability to modify the parts of the mode bits that are not related to the ACL. -- Paul B. Henson | (909) 979-6361 | http://www.csupomona.edu/~henson/ Operating Systems and Network Analyst | hen...@csupomona.edu California State Polytechnic University | Pomona CA 91768 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS Fragmentation issue - examining the ZIL
On Wed, Aug 03, 2011 at 12:32:56PM -0700, Brandon High wrote: > On Mon, Aug 1, 2011 at 4:27 PM, Daniel Carosone wrote: > > The other thing that can cause a storm of tiny IOs is dedup, and this > > effect can last long after space has been freed and/or dedup turned > > off, until all the blocks corresponding to DDT entries are rewritten. > > I wonder if this was involved here. > > Using dedup on a pool that houses an Oracle DB is Doing It Wrong in so > many ways... Indeed, but alas people still Do It Wrong. In particular, when a pool is approaching full, turning on dedup might seem like an attractive proposition to someone who doesn't understand the cost. So i just wonder if they have, or had at some time past, enabed it. -- Dan. pgpNRTzK8WULH.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Wrong rpool used after reinstall!
On 08/ 4/11 01:29 AM, Stuart James Whitefish wrote: I have Solaris on Sparc boxes available if it would help to do a net install or jumpstart. I have never done those and it looks complicated, although I think I may be able to get to the point in the u9 installer on my Intel box where it asks me whether I want to install from DVD etc. But I may be wrong, and anyway the single user shell in the u9 DVD also panics when I try to import tank so maybe that won't help. Put your old drive in a USB enclosure and connect it to another system in order to read back the data. I have only 4 sata ports on this Intel box so I have to keep pulling cables to be able to boot from a DVD and then I won't have all my drives available. I cannot move these drives to any other box because they are consumer drives and my servers all have ultras. Most modern boards will be boot from a live USB stick. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS Fragmentation issue - examining the ZIL
On Mon, Aug 1, 2011 at 4:27 PM, Daniel Carosone wrote: > The other thing that can cause a storm of tiny IOs is dedup, and this > effect can last long after space has been freed and/or dedup turned > off, until all the blocks corresponding to DDT entries are rewritten. > I wonder if this was involved here. Using dedup on a pool that houses an Oracle DB is Doing It Wrong in so many ways... -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Exapnd ZFS storage.
On Wed, Aug 3, 2011 at 3:02 AM, Nix wrote: > I have 4 disk with 1 TB of disk and I want to expand the zfs pool size. > > I have 2 more disk with 1 TB of size. > > Is it possible to expand the current RAIDz array with new disk? You can't add the new drives to your current vdev. You can create another vdev to add to your pool though. If you're adding another vdev, it should have the same geometry as your current (ie: 4 drives). The zpool command will complain if you try to add a vdev with different geometry or redundancy, though you can force it with -f. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Exapnd ZFS storage.
if you have 4 hdd then expand raidz is simple(from 4 hdd raidz) Sent from my iPad Hung-Sheng Tsao ( LaoTsao) Ph.D On Aug 3, 2011, at 3:02, Nix wrote: > Hi, > > I have 4 disk with 1 TB of disk and I want to expand the zfs pool size. > > I have 2 more disk with 1 TB of size. > > Is it possible to expand the current RAIDz array with new disk? > > Thanks, > Nix > -- > This message posted from opensolaris.org > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool import starves machine of memory
An additional data point, when i try to do a zdb -e -d and find the incomplete zfs recv snapshot I get an error as follows: # sudo zdb -e -d xxx-yy-01 | grep "%" Could not open xxx-yy-01/aaa-bb-01/aaa-bb-01-01/%1309906801, error 16 # Anyone know what error 16 means from zdb and how this might impact importing this zpool ? On Wed, Aug 3, 2011 at 9:19 AM, Paul Kraus wrote: > I am having a very odd problem, and so far the folks at Oracle > Support have not provided a working solution, so I am asking the crowd > here while still pursuing it via Oracle Support. > > The system is a T2000 running 10U9 with CPU-2010-01and two J4400 > loaded with 1 TB SATA drives. There is one zpool on the J4400 (3 x 15 > disk vdev + 3 hot spare). This system is the target for zfs send / > recv replication from our production server.The OS is UFS on local > disk. > > While I was on vacation this T2000 hung with "out of resource" > errors. Other staff tried rebooting, which hung the box. Then they > rebooted off of an old BE (10U9 without CPU-2010-01). Oracle Support > had them apply a couple patches and an IDR to address zfs "stability > and reliability problems" as well as set the following in /etc/system > > set zfs:zfs_arc_max = 0x7 (which is 28 GB) > set zfs:arc_meta_limit = 0x7 (which is 28 GB) > > The system has 32 GB RAM and 32 (virtual) CPUs. They then tried > importing the zpool and the system hung (after many hours) with the > same "out of resource" error. At this point they left the problem for > me :-( > > I removed the zfs.cache from the 10U9 + CPU 2010-10 BE and booted > from that. I then applied the IDR (IDR146118-12 )and the zfs patch it > depended on (145788-03). I did not include the zfs arc and zfs arc > meta limits as I did not think they relevant. A zpool import shows the > pool is OK and a sampling with zdb -l of the drives shows good labels. > I started importing the zpool and after many hours it hung the system > with "out of resource" errors. I had a number of tools running to see > what was going on. The only thing this system is doing is importing > the zpool. > > ARC had climbed to about 8 GB and then declined to 3 GB by the time > the system hung. This tells me that there is something else consuming > RAM and the ARC is releasing it. > > The hung TOP screen showed the largest user process only had 148 MB > allocated (and much less resident). > > VMSTAT showed a scan rate of over 900,000 (NOT a typo) and almost 8 GB > of free swap (so whatever is using memory cannot be paged out). > > So my guess is that there is a kernel module that is consuming all > (and more) of the RAM in the box. I am looking for a way to query how > much RAM each kernel module is using and script that in a loop (which > will hang when the box runs out of RAM next). I am very open to > suggestions here. > > Since this is the recv end of replication, I assume there was a zfs > recv going on at the time the system initially hung. I know there was > a 3+ TB snapshot replicating (via a 100 Mbps WAN link) when I left for > vacation, that may have still been running. I also assume that any > partial snapshots (% instead of @) are being removed when the pool is > imported. But what could be causing a partial snapshot removal, even > of a very large snapshot, to run the system out of RAM ? What caused > the initial hang of the system (I assume due to out of RAM) ? I did > not think there was a limit to the size of either a snapshot or a zfs > recv. > > Hung TOP screen: > > load averages: 91.43, 33.48, 18.989 xxx-xxx1 > 18:45:34 > 84 processes: 69 sleeping, 12 running, 1 zombie, 2 on cpu > CPU states: 95.2% idle, 0.5% user, 4.4% kernel, 0.0% iowait, 0.0% swap > Memory: 31.9G real, 199M free, 267M swap in use, 7.7G swap free > > PID USERNAME THR PR NCE SIZE RES STATE TIME FLTS CPU COMMAND > 533 root 51 59 0 148M 30.6M run 520:21 0 9.77% java > 1210 yy 1 0 0 5248K 1048K cpu25 2:08 0 2.23% xload > 14720 yy 1 59 0 3248K 1256K cpu24 1:56 0 0.03% top > 154 root 1 59 0 4024K 1328K sleep 1:17 0 0.02% vmstat > 1268 yy 1 59 0 4248K 1568K sleep 1:26 0 0.01% iostat > ... > > VMSTAT: > > kthr memory page disk faults cpu > r b w swap free re mf pi po fr de sr m0 m1 m2 m3 in sy cs us sy id > 0 0 112 8117096 211888 55 46 0 0 425 0 912684 0 0 0 0 976 166 836 0 2 98 > 0 0 112 8117096 211936 53 51 6 0 394 0 926702 0 0 0 0 976 167 833 0 2 98 > > ARC size (B): 4065882656 > > -- > {1-2-3-4-5-6-7-} > Paul Kraus > -> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ ) > -> Sound Designer: Frankenstein, A New Musical > (http://www.facebook.com/event.php?eid=123170297765140) > -> Sound Coordinator, Schenectady Light Opera Company ( > http://www.sloctheater.
Re: [zfs-discuss] Wrong rpool used after reinstall!
Thanks for your comments so far. I'll try to put everything I know into this post now that I have signed up at the forums. Solaris 10, update 8 Intel, 500G ZFS root mirror rpool. I recently received two 320G drives and realized from reading this list it would have been better if I would have done the install on the small drives but I didn't have them at the time. I added the two 320G drives and created tank mirror. I moved some data from other sources to the tank and then decided to go ahead and do a new install. In preparation for that I moved all the data I wanted to save onto the rpool mirror and then installed Solaris 10 update 8 again on the 320G drives. When my system rebooted after the installation, I saw for some reason it used my tank pool as root. I realize now since it was originally a root pool and had boot blocks this didn't help. Anyway I shut down, changed the boot order and then booted into my system. It paniced when trying to access the tank and instantly rebooted. I had to go through this several times until I caught a glimpse of one of the first messages: assertion failed: zvol_get_stats(os, nv) Here is what my system looks like when I boot into failsafe mode. # zpool import pool: rpool id: 16453600103421700325 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: rpoolONLINE mirror ONLINE c0t2d0s0 ONLINE c0t3d0s0 ONLINE pool: tank id: 12861119534757646169 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: tank ONLINE mirror ONLINE c0t0d0s0 ONLINE c0t1d0s0 ONLINE # zpool import tank cannot import 'tank': pool may be in use from other system use '-f' to import anyway When I do # zpool import -f tank I get a kernal panic and the screen flashes and the machine quickly reboots before I can see what's written, but I did see assertion failed: zvol_get_stats(os, nv) and there seems to be a patch 6801926 associated with it. I tried the following: 1. fmdump -ev. Either the command is not found when I bring my system up in the Solaris failsafe mode, or when the root pool which is ok gets mounted on /a the command is also not found. I did chroot /a /bin/sh and then the command is found but I get error messages about not having certain files available in /var, probably because I have /var on a separate filesystem. I do not know how to mount things as if the system came up normally and if I boot in the normal mode although it can import the root pool the tank pool was also a root pool at one time and it seems to mount it and then panic and reboot. I think it would be a lot easier to do whatever we can do if I could get the system to boot into normal mode without importing tank. Can anyone please explain in very clear and simple directions how to do that please? 2. Solaris 10 update 9 does boot but won't install, presumably because of not having a CD driver for my nvidia chipset, although the install does start it gives a message something like "the CD is not a Solaris OS CD".. I tried single user shell and it also panics when I zpool import -f tank. This is very discouraging because according to the Solaris 10 update 9 patch list, 6801926 is supposed to be preapplied. I don't understand why the update 8 DVD works on my system but update 9 doesn't. That is just weird. 3. Solaris 11 Express falls into maintenance mode and doesn't go any further. I get an error message something like bash: /usr/bin/hostname command not found, and I can't do anything useful. 4. OpenIndiana Server doesn't come all the way up. I get the Solaris 5.11 message and then it does nothing. 5. Milax falls into maintenance mode like Sol 11 Express and then panics when I try to import tank. I have Solaris on Sparc boxes available if it would help to do a net install or jumpstart. I have never done those and it looks complicated, although I think I may be able to get to the point in the u9 installer on my Intel box where it asks me whether I want to install from DVD etc. But I may be wrong, and anyway the single user shell in the u9 DVD also panics when I try to import tank so maybe that won't help. I have only 4 sata ports on this Intel box so I have to keep pulling cables to be able to boot from a DVD and then I won't have all my drives available. I cannot move these drives to any other box because they are consumer drives and my servers all have ultras. I have about 200G of data on the tank pool, about 100G or so I don't have anywhere else. I created this pool specifically to make a "safe place" to store data that I had accumulated over several years and didn't have organized yet. Can anyone PLEASE help me get this data back! Thank you. Jim -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http
[zfs-discuss] Exapnd ZFS storage.
Hi, I have 4 disk with 1 TB of disk and I want to expand the zfs pool size. I have 2 more disk with 1 TB of size. Is it possible to expand the current RAIDz array with new disk? Thanks, Nix -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zpool import starves machine of memory
I am having a very odd problem, and so far the folks at Oracle Support have not provided a working solution, so I am asking the crowd here while still pursuing it via Oracle Support. The system is a T2000 running 10U9 with CPU-2010-01and two J4400 loaded with 1 TB SATA drives. There is one zpool on the J4400 (3 x 15 disk vdev + 3 hot spare). This system is the target for zfs send / recv replication from our production server.The OS is UFS on local disk. While I was on vacation this T2000 hung with "out of resource" errors. Other staff tried rebooting, which hung the box. Then they rebooted off of an old BE (10U9 without CPU-2010-01). Oracle Support had them apply a couple patches and an IDR to address zfs "stability and reliability problems" as well as set the following in /etc/system set zfs:zfs_arc_max = 0x7 (which is 28 GB) set zfs:arc_meta_limit = 0x7 (which is 28 GB) The system has 32 GB RAM and 32 (virtual) CPUs. They then tried importing the zpool and the system hung (after many hours) with the same "out of resource" error. At this point they left the problem for me :-( I removed the zfs.cache from the 10U9 + CPU 2010-10 BE and booted from that. I then applied the IDR (IDR146118-12 )and the zfs patch it depended on (145788-03). I did not include the zfs arc and zfs arc meta limits as I did not think they relevant. A zpool import shows the pool is OK and a sampling with zdb -l of the drives shows good labels. I started importing the zpool and after many hours it hung the system with "out of resource" errors. I had a number of tools running to see what was going on. The only thing this system is doing is importing the zpool. ARC had climbed to about 8 GB and then declined to 3 GB by the time the system hung. This tells me that there is something else consuming RAM and the ARC is releasing it. The hung TOP screen showed the largest user process only had 148 MB allocated (and much less resident). VMSTAT showed a scan rate of over 900,000 (NOT a typo) and almost 8 GB of free swap (so whatever is using memory cannot be paged out). So my guess is that there is a kernel module that is consuming all (and more) of the RAM in the box. I am looking for a way to query how much RAM each kernel module is using and script that in a loop (which will hang when the box runs out of RAM next). I am very open to suggestions here. Since this is the recv end of replication, I assume there was a zfs recv going on at the time the system initially hung. I know there was a 3+ TB snapshot replicating (via a 100 Mbps WAN link) when I left for vacation, that may have still been running. I also assume that any partial snapshots (% instead of @) are being removed when the pool is imported. But what could be causing a partial snapshot removal, even of a very large snapshot, to run the system out of RAM ? What caused the initial hang of the system (I assume due to out of RAM) ? I did not think there was a limit to the size of either a snapshot or a zfs recv. Hung TOP screen: load averages: 91.43, 33.48, 18.989 xxx-xxx1 18:45:34 84 processes: 69 sleeping, 12 running, 1 zombie, 2 on cpu CPU states: 95.2% idle, 0.5% user, 4.4% kernel, 0.0% iowait, 0.0% swap Memory: 31.9G real, 199M free, 267M swap in use, 7.7G swap free PID USERNAME THR PR NCE SIZE RES STATE TIME FLTSCPU COMMAND 533 root 51 59 0 148M 30.6M run 520:210 9.77% java 1210 yy 1 0 0 5248K 1048K cpu25 2:080 2.23% xload 14720 yy 1 59 0 3248K 1256K cpu24 1:560 0.03% top 154 root 1 59 0 4024K 1328K sleep 1:170 0.02% vmstat 1268 yy 1 59 0 4248K 1568K sleep 1:260 0.01% iostat ... VMSTAT: kthr memorypagedisk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m2 m3 in sy cs us sy id 0 0 112 8117096 211888 55 46 0 0 425 0 912684 0 0 0 0 976 166 836 0 2 98 0 0 112 8117096 211936 53 51 6 0 394 0 926702 0 0 0 0 976 167 833 0 2 98 ARC size (B): 4065882656 -- {1-2-3-4-5-6-7-} Paul Kraus -> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ ) -> Sound Designer: Frankenstein, A New Musical (http://www.facebook.com/event.php?eid=123170297765140) -> Sound Coordinator, Schenectady Light Opera Company ( http://www.sloctheater.org/ ) -> Technical Advisor, RPI Players ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Saving data across install
Please ignore this post. Bad things happened and now there is another thread for it. Thank you. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Wrong rpool used after reinstall!
You wrote: > > > > Hi Roy, things got alot worse since my first email. I don't know what > > happened but I can't import the old pool at all. It shows no errors but when > > I import it I get a kernel panic from assertion failed: zvol_get_stats(os, > > nv) which looks like is fixed by patch 6801926 which is applied in Solaris > > 10u9. But I cannot boot update 9 on this box! I tried Solaris Express, none > > of those will run right either. They all go into maintenance mode. The last > > thing I can boot is update 8 and that is the one with the ZFS bug. > > If they go into maintenance mode but could recognize the disk, you > should still be able to do zfs stuff (zpool import, etc). If you're > lucky you'd only miss the GUI Thank you for your comments. This is pretty frustrating. Unfortunately I'm hitting another bug I saw on the net. Once the Express Live CD or text installer falls back to maintenance mode, I keep getting a msg bash: /usr/bin/hostname command not found (from memory, may not be exact). All ZFS commands fail with this message. I don't know what causes this and I'm surprised since Solaris 10 update 8 works mostly fine on the same hardware. I would expect support to get wider not less but the opposite seems to be happening because I can't install update 9 on this machine. > > I have 200G of files I deliberately copied to this new mirror and now I > > can't get at them! Any ideas? > > Another thing you can try (albeit more complex) is use another OS > (install, or even use a linux Live CD), install virtualbox or similar > on it, pass the disk as raw vmdk, and use solaris express on the VM. > You should be able to at least import the pool and recover the data. I didn't think I would be able to use raw disks with exported pools in a VM but your comment is interesting. I will consider it. The host is bootable, it just immediately panics upon trying to import the 2nd pool. Is there some way I can force a normal boot where my root pool is mounted but it doesn't mount the other pool? If so I could install VirtualBox and try your suggestion without moving the drives to another box. Aren't there any ZFS recovery tools, or is ZFS just not expected to break? I mentioned I can boot the update 9 installer, but it fails when trying to read media from the DVD because of a lack of nvidia drivers (documented limitation) I wonder if I can do some kind of network or jumpstart install. I have no other Solaris Intel boxes (and this post should explain some of the reasons why) but I have several Solaris Sparc boxes. I haven't gone through the doc on net/jumpstart installs, there is alot to read. But maybe this would get update 9 on that box and maybe it could import the pool. Are there any other Solaris based live CD or DVD I could try that have known, good ZFS support? I will need something where I can scp/rcp/rsync the data off that box, assuming that I can import it somehow. Thanks, Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss