Re: [zfs-discuss] million files in single directory
On Sat, 3 Oct 2009, Jeff Haferman wrote: When I go into directory 0, it takes about a minute for an ls -1 | grep wc to return (it has about 12,000 files). Directory 1 takes between 5-10 minutes for the same command to return (it has about 50,000 files). This seems kind of slow. In the directory with a million files that I keep around for testing this is the time for the first access: % time \ls -1 | grep wc \ls -1 4.70s user 1.20s system 32% cpu 17.994 total grep wc 0.11s user 0.02s system 0% cpu 17.862 total and for the second access: % time \ls -1 | grep wc \ls -1 4.66s user 1.17s system 69% cpu 8.366 total grep wc 0.11s user 0.02s system 1% cpu 8.234 total However, my directory was created as quickly as possible rather than incrementally over a long period of time so it lacks the longer/increased disks seeks caused by fragmentation and block allocations. That said, directories with 50K files list quite quickly here. I did an rsync of this directory structure to another filesystem [lustre-based, FWIW] and it took about 24 hours to complete. We have Rsync is very slow in such situations. What version of Solaris are you using? The Solaris version (including patch version if using Solaris 10) can make a big difference. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Best way to convert checksums
re == Richard Elling richard.ell...@gmail.com writes: re The probability of the garbage having both a valid fletcher2 re checksum at the proper offset and having the proper sequence re number and having the right log chain link and having the re right block size is considerably lower than the weakness of re fletcher2. I'm having trouble parsing this. I think you're confusing a few different failure modes: * ZIL entry is written, but corrupted by the storage, so that, for example, an entry should be read from the mirrored ZIL instead. + broken fletcher2 detects the storage corruption CASE A: Good! + broken fletcher2 misses the error, so that corrupted data is replayed from ZIL into the proper pool, possibly adding a stronger checksum to the corrupt data while writing it. CASE B: Bad! + broken fletcher2 misinterprets storage corruption as signalling the end of the ZIL, and any data in the ZIL after the corrupt entry is truncated without even attempting to read the mirror. (does this happen?) CASE C: Bad! * ZIL entry is intentional garbage, either a partially-written entry or an old entry, and should be treated as the end of the ZIL + broken fletcher2 identifies the partially written entry by a checksum mismatch, or the sequence number identifies it as old CASE D: Good! + broken fletcher2 misidentifies a partially-written entry as complete because of a hash collision CASE E: Bad! + (hypothetical, only applies to non-existent fixed system) working fletcher2 or broken-good-enough fletcher4 misidentifies a partially-written entry as complete because of a hash collision CASE F: Bad! If I read your sentence carefully and try to match it with this chart, it seems like you're saying P(CASE F) P(CASE E), which seems like an argument for fixing the checksum. While you don't say so, I presume from your other posts you're trying to make a case for doing nothing, so I'm confused. I was mostly thinking about CASE B though. It seems like the special way the ZIL works has nothing to do with CASE B: if you send data through the ZIL to a sha256 pool, it can be written to ZIL under broken-fletcher2, corrupted by the storage, and then read in and played back corrupt but covered with a sha256 checksum to the pool proper. AFAICT your relative-probability sentence has nothing to do with CASE B. re Unfortunately, the ZIL is also latency sensitive, so the re performance case gets stronger The performance case advocating what? not fixing the broken checksum? re while the additional error checking already boosts the re dependability case. what additional error checking? Isn't the whole specialness of the ZIL that the checksum is needed in normal operation, absent storage subsystem corruption, as I originally said? It seems like the checksum's strength is more important here, not less. pgpMTzwhPNdUa.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZPOOL Metadata / Data Error - Help
Hi all ! I have a serious problem, with a server, and i'm hoping that some one could help me how to understand what's wrong. So basically i have a server with a pool of 6 disks, and after a zpool scrub i go the message : errors: Permanent errors have been detected in the following files: metadata:0x0 metadata:0x15 The version of the opensolaris is 5.11 snv_101b (yes, i now, quite old). This server has been up and running for more than 4 months, with weekly zpool scrubs, and now i got this message. Here are some extra details about the system: 1 - i can still access the data in the pool , but i don't know if i can access all the data and/or if all the data is not corrupted 2 - nothing was changed in the hardware 3 - all the disks are ST31000340NS-SN06 , Seagate 1TB 7.200 rpm enterprise class , firmware SN06 4 - all the disks are connected to a LSI Logic SAS1068E connected to a JBOD chassis (Supermicro) 5 - the server is a SUN X2200 Dual-Core 6 - using the lsiutil, and querying the Display phy counters i see : Expander (Handle 0009) Phy 21: Link Up Invalid DWord Count 1,171 Running Disparity Error Count 937 Loss of DWord Synch Count 0 Phy Reset Problem Count 0 Expander (Handle 0009) Phy 22: Link Up Invalid DWord Count 2,110,435 Running Disparity Error Count 855,781 Loss of DWord Synch Count 3 Phy Reset Problem Count 0 Expander (Handle 0009) Phy 23: Link Up Invalid DWord Count 740,029 Running Disparity Error Count 716,196 Loss of DWord Synch Count 1 Phy Reset Problem Count 0 Expander (Handle 0009) Phy 24: Link Up Invalid DWord Count 705,870 Running Disparity Error Count 692,280 Loss of DWord Synch Count 1 Phy Reset Problem Count 0 Expander (Handle 0009) Phy 25: Link Up Invalid DWord Count 698,935 Running Disparity Error Count 667,148 Loss of DWord Synch Count 1 Phy Reset Problem Count 0 7 - the /var/log/messages show o SCSI transport failed: reason 'reset': retrying command o SCSI transport failed: reason 'reset': giving up Maybe i'm wrong...but it seems like the disks started to report errors? The reason behind the fact that i don't know if all the data is accessible and valid is because the pool size is quite big, as seen : NAME SIZE USED AVAILCAP HEALTH ALTROOT POOL01 2.72T 1.71T 1.01T62% ONLINE - It might be the fact that i have been suffering from this problem from some time, but the lsi hba had never reported any error, and i assumed that ZFS was build to deal with this kind of problems : the silent data corruption . I'm would to understand if the problems started due to a high load in the LSI hba that lead to timeouts and therefore disk errors, of if the the LSI hba opensolaris driver was overloaded ,resulting in disk errors and LSI hba errors... Any clue to see what lead to what? Even more importand did i lost data, or zfs is reporting errors to disk drivers errors, but the data already existing is okay, and the new data may be affected? Is the zpool metadata recoverable? My biggest concern, is to know if my pool is corrupted, and if so how can i fix the zpool,metadata, problem. Thanks for all your time, Bruno r...@server01:/# zpool status -vx pool: POOL01 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM POOL01 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t9d0 ONLINE 0 0 0 c5t10d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t11d0 ONLINE 0 0 0 c5t12d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t13d0 ONLINE 0 0 0 c5t14d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: metadata:0x0 metadata:0x15 -- This message has been scanned for viruses and dangerous
Re: [zfs-discuss] ZPOOL Metadata / Data Error - Help
Bruno Sousa wrote: Action: Restore the file in question if possible. Otherwise restore the entire pool from backup. metadata:0x0 metadata:0x15 Hmm, and what file(s) would this be? -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS 10u7 5/09 | OpenSolaris 2010.02 b123 + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZPOOL Metadata / Data Error - Help
Action: Restore the file in question if possible. Otherwise restore the entire pool from backup. metadata:0x0 metadata:0x15 bet its in a snapshot that looks to have been destroyed already. try zpool clear POOL01 zpool scrub POOL01 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] .zfs snapshots on subdirectories?
Edward If you look at the man page:- snapshot A read-only version of a file system or volume at a given point in time. It is specified as filesys...@name or vol...@name. I think you've taken volume snapshots. I believe you need to make file system snapshots and each users/username a zfs file system. Lets play.. r...@norton:~# zpool create -f storagepool c9t5d0 r...@norton:~# zfs create storagepool/users r...@norton:~# zfs create storagepool/users/bob r...@norton:~# zfs create storagepool/users/dick r...@norton:# cd /storagepool/users/bob r...@norton:# touch foo r...@norton:# zfs snapshot storagepool/users/b...@now r...@norton# ls -alR /storagepool/users/bob/.zfs /storagepool/users/bob/.zfs: total 3 dr-xr-xr-x 4 root root 4 2009-10-05 12:09 . drwxr-xr-x 2 root root 3 2009-10-05 12:14 .. dr-xr-xr-x 2 root root 2 2009-10-05 12:09 shares dr-xr-xr-x 2 root root 2 2009-10-05 12:09 snapshot /storagepool/users/bob/.zfs/shares: total 2 dr-xr-xr-x 2 root root 2 2009-10-05 12:09 . dr-xr-xr-x 4 root root 4 2009-10-05 12:09 .. /storagepool/users/bob/.zfs/snapshot: total 2 dr-xr-xr-x 2 root root 2 2009-10-05 12:09 . dr-xr-xr-x 4 root root 4 2009-10-05 12:09 .. drwxr-xr-x 2 root root 3 2009-10-05 12:14 now /storagepool/users/bob/.zfs/snapshot/now: total 2 drwxr-xr-x 2 root root 3 2009-10-05 12:14 . dr-xr-xr-x 3 root root 3 2009-10-05 12:09 .. -rw-r--r-- 1 root root 0 2009-10-05 12:14 foo If you want a .zfs in /storagepool/users/eharvey/some/foo/dir it needs to be a separate file system. Edward Ned Harvey wrote: Suppose I have a storagepool: /storagepool And I have snapshots on it. Then I can access the snaps under /storagepool/.zfs/snapshots But is there any way to enable this within all the subdirs? For example, cd /storagepool/users/eharvey/some/foo/dir cd .zfs I don’t want to create a new filesystem for every subdir. I just want to automatically have the “.zfs” hidden directory available within all the existing subdirs, if that’s possible. Thanks…. www.eagle.co.nz This email is confidential and may be legally privileged. If received in error please destroy and immediately notify us. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZPOOL data corruption - Help
Hi all ! I have a serious problem, with a server, and i'm hoping that some one could help me how to understand what's wrong. So basically i have a server with a pool of 6 disks, and after a zpool scrub i go the message : errors: Permanent errors have been detected in the following files: metadata:0x0 metadata:0x15 The version of the opensolaris is 5.11 snv_101b (yes, i now, quite old). This server has been up and running for more than 4 months, with weekly zpool scrubs, and now i got this message. Here are some extra details about the system: 1 - i can still access the data in the pool , but i don't know if i can access all the data and/or if all the data is not corrupted 2 - nothing was changed in the hardware 3 - all the disks are ST31000340NS-SN06 , Seagate 1TB 7.200 rpm enterprise class , firmware SN06 4 - all the disks are connected to a LSI Logic SAS1068E connected to a JBOD chassis (Supermicro) 5 - the server is a SUN X2200 Dual-Core 6 - using the lsiutil, and querying the Display phy counters i see : Expander (Handle 0009) Phy 21: Link Up Invalid DWord Count 1,171 Running Disparity Error Count 937 Loss of DWord Synch Count 0 Phy Reset Problem Count 0 Expander (Handle 0009) Phy 22: Link Up Invalid DWord Count 2,110,435 Running Disparity Error Count 855,781 Loss of DWord Synch Count 3 Phy Reset Problem Count 0 Expander (Handle 0009) Phy 23: Link Up Invalid DWord Count 740,029 Running Disparity Error Count 716,196 Loss of DWord Synch Count 1 Phy Reset Problem Count 0 Expander (Handle 0009) Phy 24: Link Up Invalid DWord Count 705,870 Running Disparity Error Count 692,280 Loss of DWord Synch Count 1 Phy Reset Problem Count 0 Expander (Handle 0009) Phy 25: Link Up Invalid DWord Count 698,935 Running Disparity Error Count 667,148 Loss of DWord Synch Count 1 Phy Reset Problem Count 0 7 - the /var/log/messages show o SCSI transport failed: reason 'reset': retrying command o SCSI transport failed: reason 'reset': giving up Maybe i'm wrong...but it seems like the disks started to report errors? The reason behind the fact that i don't know if all the data is accessible is because the pool size is quite big, as seen : NAME SIZE USED AVAILCAP HEALTH ALTROOT POOL01 2.72T 1.71T 1.01T62% ONLINE - It might be the fact that i have been suffering from this problem from some time, but the lsi hba had never reported any error, and i assumed that ZFS was build to deal with this kind of problems : the silent data corruption . I'm would to understand if the problems started due to a high load in the LSI hba that lead to timeouts and therefore disk errors, of if the the LSI hba opensolaris driver was overloaded ,resulting in disk errors and LSI hba errors... Any clue to see what lead to what? Even more importand did i lost data, or zfs is reporting errors to disk drivers errors, but the data already existing is okay, and the new data may be affected? Is the zpool metadata recoverable? My biggest concern, is to know if my pool is corrupted, and if so how can i fix the zpool,metadata, problem. Thanks for all your time, Bruno r...@server01:/# zpool status -vx pool: POOL01 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM POOL01 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t9d0 ONLINE 0 0 0 c5t10d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t11d0 ONLINE 0 0 0 c5t12d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t13d0 ONLINE 0 0 0 c5t14d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: metadata:0x0 metadata:0x15 -- This message has been
Re: [zfs-discuss] Best way to convert checksums
On Oct 4, 2009, at 11:51 AM, Miles Nordin wrote: re == Richard Elling richard.ell...@gmail.com writes: re The probability of the garbage having both a valid fletcher2 re checksum at the proper offset and having the proper sequence re number and having the right log chain link and having the re right block size is considerably lower than the weakness of re fletcher2. I'm having trouble parsing this. I think you're confusing a few different failure modes: * ZIL entry is written, but corrupted by the storage, so that, for example, an entry should be read from the mirrored ZIL instead. This is attempted, if you have a mirrored slog. + broken fletcher2 detects the storage corruption CASE A: Good! + broken fletcher2 misses the error, so that corrupted data is replayed from ZIL into the proper pool, possibly adding a stronger checksum to the corrupt data while writing it. CASE B: Bad! + broken fletcher2 misinterprets storage corruption as signalling the end of the ZIL, and any data in the ZIL after the corrupt entry is truncated without even attempting to read the mirror. (does this happen?) CASE C: Bad! * ZIL entry is intentional garbage, either a partially-written entry or an old entry, and should be treated as the end of the ZIL + broken fletcher2 identifies the partially written entry by a checksum mismatch, or the sequence number identifies it as old CASE D: Good! If the checksum mismatches, you can't go any further because the pointer to the next ZIL log entry cannot be trusted. So the roll forward stops. This is how such logs work -- there is no end-of-log record. + broken fletcher2 misidentifies a partially-written entry as complete because of a hash collision CASE E: Bad! + (hypothetical, only applies to non-existent fixed system) working fletcher2 or broken-good-enough fletcher4 misidentifies a partially-written entry as complete because of a hash collision CASE F: Bad! As I said before, if the checksum matches, then the data is checked for sequence number = previous + 1, the blk_birth == 0, and the size is correct. Since this data lives inside the block, it is unlikely that a collision would also result in a valid block. In other words, ZFS doesn't just trust the checksum for slog entries. -- richard If I read your sentence carefully and try to match it with this chart, it seems like you're saying P(CASE F) P(CASE E), which seems like an argument for fixing the checksum. While you don't say so, I presume from your other posts you're trying to make a case for doing nothing, so I'm confused. I was mostly thinking about CASE B though. It seems like the special way the ZIL works has nothing to do with CASE B: if you send data through the ZIL to a sha256 pool, it can be written to ZIL under broken-fletcher2, corrupted by the storage, and then read in and played back corrupt but covered with a sha256 checksum to the pool proper. AFAICT your relative-probability sentence has nothing to do with CASE B. re Unfortunately, the ZIL is also latency sensitive, so the re performance case gets stronger The performance case advocating what? not fixing the broken checksum? re while the additional error checking already boosts the re dependability case. what additional error checking? Isn't the whole specialness of the ZIL that the checksum is needed in normal operation, absent storage subsystem corruption, as I originally said? It seems like the checksum's strength is more important here, not less. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] .zfs snapshots on subdirectories?
OOPS just spotted you said you don't want a FS for each sub-dir :-) Trevor Pretty wrote: Edward If you look at the man page:- snapshot A read-only version of a file system or volume at a given point in time. It is specified as filesys...@name or vol...@name. I think you've taken volume snapshots. I believe you need to make file system snapshots and each users/username a zfs file system. Lets play.. r...@norton:~# zpool create -f storagepool c9t5d0 r...@norton:~# zfs create storagepool/users r...@norton:~# zfs create storagepool/users/bob r...@norton:~# zfs create storagepool/users/dick r...@norton:# cd /storagepool/users/bob r...@norton:# touch foo r...@norton:# zfs snapshot storagepool/users/b...@now r...@norton# ls -alR /storagepool/users/bob/.zfs /storagepool/users/bob/.zfs: total 3 dr-xr-xr-x 4 root root 4 2009-10-05 12:09 . drwxr-xr-x 2 root root 3 2009-10-05 12:14 .. dr-xr-xr-x 2 root root 2 2009-10-05 12:09 shares dr-xr-xr-x 2 root root 2 2009-10-05 12:09 snapshot /storagepool/users/bob/.zfs/shares: total 2 dr-xr-xr-x 2 root root 2 2009-10-05 12:09 . dr-xr-xr-x 4 root root 4 2009-10-05 12:09 .. /storagepool/users/bob/.zfs/snapshot: total 2 dr-xr-xr-x 2 root root 2 2009-10-05 12:09 . dr-xr-xr-x 4 root root 4 2009-10-05 12:09 .. drwxr-xr-x 2 root root 3 2009-10-05 12:14 now /storagepool/users/bob/.zfs/snapshot/now: total 2 drwxr-xr-x 2 root root 3 2009-10-05 12:14 . dr-xr-xr-x 3 root root 3 2009-10-05 12:09 .. -rw-r--r-- 1 root root 0 2009-10-05 12:14 foo If you want a .zfs in /storagepool/users/eharvey/some/foo/dir it needs to be a separate file system. Edward Ned Harvey wrote: Suppose I have a storagepool: /storagepool And I have snapshots on it. Then I can access the snaps under /storagepool/.zfs/snapshots But is there any way to enable this within all the subdirs? For example, cd /storagepool/users/eharvey/some/foo/dir cd .zfs I don’t want to create a new filesystem for every subdir. I just want to automatically have the “.zfs” hidden directory available within all the existing subdirs, if that’s possible. Thanks…. www.eagle.co.nz This email is confidential and may be legally privileged. If received in error please destroy and immediately notify us. -- Trevor Pretty | Technical Account Manager | +64 9 639 0652 | +64 21 666 161 Eagle Technology Group Ltd. Gate D, Alexandra Park, Greenlane West, Epsom Private Bag 93211, Parnell, Auckland www.eagle.co.nz This email is confidential and may be legally privileged. If received in error please destroy and immediately notify us. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] bigger zfs arc
On Sat, Oct 3, 2009 at 11:33 AM, Richard Elling richard.ell...@gmail.comwrote: On Oct 3, 2009, at 10:26 AM, Chris Banal wrote: On Fri, Oct 2, 2009 at 10:57 PM, Richard Elling richard.ell...@gmail.com wrote: c is the current size the ARC. c will change dynamically, as memory pressure and demand change. How is the relative greediness of c determined? Is there a way to make it more greedy on systems with lots of free memory? AFAIK, there is no throttle on the ARC, so c will increase as the I/O demand dictates. The L2ARC has a fill throttle because those IOPS can compete with the other devices on the system. Other then memory pressure what would cause c to decrease? On a system that does nightly backups which are many times the amount of physical memory and does nothing but nfs. Why would we see c well below zfs_arc_max and plenty of free memory? Thanks, Chris ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] million files in single directory
Rob Logan wrote: Directory 1 takes between 5-10 minutes for the same command to return (it has about 50,000 files). That said, directories with 50K files list quite quickly here. a directory with 52,705 files lists in half a second here 36 % time \ls -1 /dev/null 0.41u 0.07s 0:00.50 96.0% perhaps your ARC is too small? I set it according to Section 1.1 of the ZFS Evil Tuning Guide: http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Move zpool from one controller to another, just swap cables?
Hi, I've been having some serious problems with my RaidZ2 array since I updated to 2009.06 on friday (from 2008.05). Its 10 drives with 1 hot spare with 8 drives on a SAS card and 3 drives on the motherboards SATA connectors. I'm worried that the SAS card is either malfunctioning or 2009.6 is having issues with it so I want to switch over to my 8 port Areca raid card but can I simply export the array, shut down the computer and move the 8 cables from the 3ware to the areca, power back up and import the array again? Thanks, Sam -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Move zpool from one controller to another, just swap cables?
On Mon, Oct 5, 2009 at 10:04 AM, Sam s...@smugmug.com wrote: Hi, I've been having some serious problems with my RaidZ2 array since I updated to 2009.06 on friday (from 2008.05). Its 10 drives with 1 hot spare with 8 drives on a SAS card and 3 drives on the motherboards SATA connectors. I'm worried that the SAS card is either malfunctioning or 2009.6 is having issues with it so I want to switch over to my 8 port Areca raid card but can I simply export the array, shut down the computer and move the 8 cables from the 3ware to the areca, power back up and import the array again? Since it's not the root pool, then yes. If it's root pool, it'd involve booting with live CD (at least that's what I had to do with opensolaris on Xen). -- Fajar ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
