Re: [zfs-discuss] 3510 - some new tests
So this is the interesting data , right ? 1. 3510, RAID-10 using 24 disks from two enclosures, random optimization, 32KB stripe width, write-back, one LUN 1.1 filebench/varmail for 60s a. ZFS on top of LUN, atime=off IO Summary: 490054 ops 8101.6 ops/s, (1246/1247 r/w) 39.9mb/s, 291us cpu/op, 6.1ms latency 2. 3510, 2x (4x RAID-0(3disks)), 32KB stripe width, random optimization, write back. 4 R0 groups are in one enclosure and assigned to primary controller then another 4 R0 groups are in other enclosure and assigned to secondary controller. Then RAID-10 is created with mirror groups between controllers. 24x disks total as in #1. 2.1 filebench/varmail 60s a. ZFS RAID-10, atime=off IO Summary: 379284 ops 6273.4 ops/s, (965/965 r/w) 30.9mb/s, 314us cpu/op, 8.0ms latency Have you tried 1M stripes spacially in case 2 ? -r ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] unaccounted for daily growth in ZFS disk space usage
We finally flipped the switch on one of our ZFS-based servers, with approximately 1TB of 2.8TB (3 stripes of 950MB or so, each of which is a RAID5 volume on the adaptec card). We have snapshots every 4 hours for the first few days. If you add up the snapshot references it appears somewhat high versus daily use (mostly mail boxes, spam, etc changing), but say an aggregate of no more than 400+MB a day. However, zfs list shows our daily pool as a whole, and per day we are growing by .01TB, or more specifically 80GB a day. That's a far cry different than the 400MB we can account for. Is it possible that metadata/ditto blocks, or the like is trully growing that rapidly. By our calculations, we will triple our disk space (sitting still) in 6 months and use up the remaining 1.7TB. Of course, this is only with 2-3 days of churn, but its an alarming rate where before on the NetApp we didn't see anything close to this rate. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re[2]: [zfs-discuss] 3510 - some new tests
Hello Roch, Thursday, August 24, 2006, 3:37:34 PM, you wrote: R> So this is the interesting data , right ? R> 1. 3510, RAID-10 using 24 disks from two enclosures, random R> optimization, 32KB stripe width, write-back, one LUN R> 1.1 filebench/varmail for 60s R> a. ZFS on top of LUN, atime=off R> IO Summary: 490054 ops 8101.6 ops/s, (1246/1247 r/w) R> 39.9mb/s,291us cpu/op, 6.1ms latency R> 2. 3510, 2x (4x RAID-0(3disks)), 32KB stripe width, R> random optimization, write back. 4 R0 groups are in one enclosure R> and assigned to primary controller then another 4 R0 groups are in other enclosure R> and assigned to secondary controller. Then RAID-10 is created with R> mirror groups between controllers. 24x disks total as in #1. R>2.1 filebench/varmail 60s R> a. ZFS RAID-10, atime=off R> IO Summary: 379284 ops 6273.4 ops/s, (965/965 r/w) R> 30.9mb/s,314us cpu/op, 8.0ms latency R> Have you tried 1M stripes spacially in case 2 ? R> -r I did try with 128KB and 256KB stripe width - the same results (difference less than 5%). I haven't tested 1MB 'coz maximum for 3510 is 256KB. -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zpool hangs
Hi. S10U2 + patches, SPARC, Generic_118833-20 I issued zpool create but possibly some (or all) MPxIO devices aren't there anymore. Now I can't kill zpool. bash-3.00# zpool create f3-1 mirror c5t600C0FF0098FD5275268D600d0 c5t600C0FF0098FD564175B0600d0 mirror c5t600C0FF0098FD567D3965E00d0 c5t600C0FF0098FD57E58FAEB00d0 mirror c5t600C0FF0098FD50642D41000d0 c5t600C0FF0098FD5580A39C100d0 mirror c5t600C0FF0098FD53F34388300d0 c5t600C0FF0098FD57A96A41900d0 ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C bash-3.00# ps -ef|grep zpool root 2516 2409 0 16:23:02 pts/6 0:00 zpool create f3-1 mirror c5t600C0FF0098FD5275268D600d0 c5t600C0FF00 bash-3.00# kill -9 2516 bash-3.00# kill -9 2516 bash-3.00# kill -9 2516 bash-3.00# ps -ef|grep zpool root 2516 2409 0 16:23:02 pts/6 0:00 zpool create f3-1 mirror c5t600C0FF0098FD5275268D600d0 c5t600C0FF00 bash-3.00# pstack 2516 pstack: cannot examine 2516: no such process bash-3.00# bash-3.00# mdb -kw Loading modules: [ unix krtld genunix dtrace specfs ufs sd md ip sctp usba fcp fctl qlc ssd lofs zfs random logindmux ptm cpc fcip crypto nfs ipc ] > ::ps!grep pool R 2516 2409 2516 2403 0 0x4a304902 06001f3c8410 zpool > 06001f3c8410::walk thread|::findstack -v stack pointer for thread 3000352e660: 2a1041f0b51 [ 02a1041f0b51 sema_p+0x130() ] 02a1041f0c01 biowait+0x6c(60017fc1e80, 0, 183d400, 180c000, 790, 60017fc1e80) 02a1041f0cb1 ssd_send_scsi_cmd+0x394(760790, 2a1041f1668, 600018ef580, 1, 1, 0) 02a1041f0da1 ssd_send_scsi_TEST_UNIT_READY+0x100(600018ef580, 1, f2, 790, 0, 0) 02a1041f0eb1 ssd_get_media_info+0x64(760790, ffbfa8c8, 15, 600018ef580, 790, 8) 02a1041f0fc1 ssdioctl+0xb28(760790, 198b2800, 600018ef580, 0, 60006f20860, 2a1041f1adc) 02a1041f10d1 fop_ioctl+0x20(60007bae340, 42a, ffbfa8c8, 15, 60006f20860, 11ff9f0) 02a1041f1191 ioctl+0x184(3, 60007a14a88, ffbfa8c8, fff8, 73, 42a) 02a1041f12e1 syscall_trap32+0xcc(3, 42a, ffbfa8c8, fff8, 73, 70) stack pointer for thread 3fe0c80: 2a104208f11 [ 02a104208f11 cv_wait+0x38() ] 02a104208fc1 exitlwps+0x11c(0, 3fe0c80, 4a004002, 6001f3c8410, 20a0, 4a004002) 02a104209071 proc_exit+0x20(2, 2, 0, 6efc, 6000787b1a8, 1857400) 02a104209121 exit+8(2, 2, , 0, 6001f3c8410, 2) 02a1042091d1 post_syscall+0x3e8(4, 0, 3fe0e54, c9, 6000787b1a8, 1) 02a1042092e1 syscall_trap32+0x18c(0, 0, 0, 0, fed7bfa0, 4) > This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re[3]: [zfs-discuss] 3510 - some new tests
Hello Robert, Thursday, August 24, 2006, 4:25:16 PM, you wrote: RM> Hello Roch, RM> Thursday, August 24, 2006, 3:37:34 PM, you wrote: R>> So this is the interesting data , right ? R>> 1. 3510, RAID-10 using 24 disks from two enclosures, random R>> optimization, 32KB stripe width, write-back, one LUN R>> 1.1 filebench/varmail for 60s R>> a. ZFS on top of LUN, atime=off R>> IO Summary: 490054 ops 8101.6 ops/s, (1246/1247 r/w) R>> 39.9mb/s,291us cpu/op, 6.1ms latency R>> 2. 3510, 2x (4x RAID-0(3disks)), 32KB stripe width, R>> random optimization, write back. 4 R0 groups are in one enclosure R>> and assigned to primary controller then another 4 R0 groups are in other enclosure R>> and assigned to secondary controller. Then RAID-10 is created with R>> mirror groups between controllers. 24x disks total as in #1. R>>2.1 filebench/varmail 60s R>> a. ZFS RAID-10, atime=off R>> IO Summary: 379284 ops 6273.4 ops/s, (965/965 r/w) R>> 30.9mb/s,314us cpu/op, 8.0ms latency R>> Have you tried 1M stripes spacially in case 2 ? R>> -r RM> I did try with 128KB and 256KB stripe width - the same results RM> (difference less than 5%). RM> I haven't tested 1MB 'coz maximum for 3510 is 256KB. I've just tested with 4k - the same result. -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS and very large directories
Due to legacy constraints, I have a rather complicated system that is currently using Sun QFS (actually the SAM portion of it.) For a lot of reasons, I'd like to look at moving to ZFS, but would like a "sanity check" to make sure ZFS is suitable to this application. First of all, we are NOT using the cluster capabilities of SAMFS. Instead, we're using it as a way of dealing with one directory that contains approximately 100,000 entries. The question is this: I know from the specs that ZFS can handle a directory with this many entries, but what I'm actually wondering is how directory lookups are handled? That is, if I do a "cd foo05" in a directory with foo01 through foo10, will the filesystem have to scroll through all the directory contents to find foo05, or does it use a btree or something to handle this? This directory is, in turn, shared out over NFS. Are there any issues I should be aware of with this sort of installation? Thanks for any advice or input! Patrick Narkinsky Sr. System Engineer EDS This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re[4]: [zfs-discuss] 3510 - some new tests
Hello Robert, Thursday, August 24, 2006, 4:44:26 PM, you wrote: RM> Hello Robert, RM> Thursday, August 24, 2006, 4:25:16 PM, you wrote: RM>> Hello Roch, RM>> Thursday, August 24, 2006, 3:37:34 PM, you wrote: R>>> So this is the interesting data , right ? R>>> 1. 3510, RAID-10 using 24 disks from two enclosures, random R>>> optimization, 32KB stripe width, write-back, one LUN R>>> 1.1 filebench/varmail for 60s R>>> a. ZFS on top of LUN, atime=off R>>> IO Summary: 490054 ops 8101.6 ops/s, (1246/1247 r/w) R>>> 39.9mb/s,291us cpu/op, 6.1ms latency R>>> 2. 3510, 2x (4x RAID-0(3disks)), 32KB stripe width, R>>> random optimization, write back. 4 R0 groups are in one enclosure R>>> and assigned to primary controller then another 4 R0 groups are in other enclosure R>>> and assigned to secondary controller. Then RAID-10 is created with R>>> mirror groups between controllers. 24x disks total as in #1. R>>>2.1 filebench/varmail 60s R>>> a. ZFS RAID-10, atime=off R>>> IO Summary: 379284 ops 6273.4 ops/s, (965/965 r/w) R>>> 30.9mb/s,314us cpu/op, 8.0ms latency R>>> Have you tried 1M stripes spacially in case 2 ? R>>> -r RM>> I did try with 128KB and 256KB stripe width - the same results RM>> (difference less than 5%). RM>> I haven't tested 1MB 'coz maximum for 3510 is 256KB. RM> I've just tested with 4k - the same result. And now I tried with creating two stripes on 3510, each with 12 disks, 32KB stripe width, each on different controller. Then using ZFS I mirrored them. And the result is ~6300 IOPS for the same test. Looks like I'll go with HW raid and zfs as file system for several reasons. -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool hangs
Robert, One of your disks is not responding. I've been trying to track down why the scsi command is not being timed out but for now check out each of the devices to make sure they are healthy. BTW, if you capture a corefile let me know. Thanks, George Robert Milkowski wrote: Hi. S10U2 + patches, SPARC, Generic_118833-20 I issued zpool create but possibly some (or all) MPxIO devices aren't there anymore. Now I can't kill zpool. bash-3.00# zpool create f3-1 mirror c5t600C0FF0098FD5275268D600d0 c5t600C0FF0098FD564175B0600d0 mirror c5t600C0FF0098FD567D3965E00d0 c5t600C0FF0098FD57E58FAEB00d0 mirror c5t600C0FF0098FD50642D41000d0 c5t600C0FF0098FD5580A39C100d0 mirror c5t600C0FF0098FD53F34388300d0 c5t600C0FF0098FD57A96A41900d0 ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C bash-3.00# ps -ef|grep zpool root 2516 2409 0 16:23:02 pts/6 0:00 zpool create f3-1 mirror c5t600C0FF0098FD5275268D600d0 c5t600C0FF00 bash-3.00# kill -9 2516 bash-3.00# kill -9 2516 bash-3.00# kill -9 2516 bash-3.00# ps -ef|grep zpool root 2516 2409 0 16:23:02 pts/6 0:00 zpool create f3-1 mirror c5t600C0FF0098FD5275268D600d0 c5t600C0FF00 bash-3.00# pstack 2516 pstack: cannot examine 2516: no such process bash-3.00# bash-3.00# mdb -kw Loading modules: [ unix krtld genunix dtrace specfs ufs sd md ip sctp usba fcp fctl qlc ssd lofs zfs random logindmux ptm cpc fcip crypto nfs ipc ] ::ps!grep pool R 2516 2409 2516 2403 0 0x4a304902 06001f3c8410 zpool 06001f3c8410::walk thread|::findstack -v stack pointer for thread 3000352e660: 2a1041f0b51 [ 02a1041f0b51 sema_p+0x130() ] 02a1041f0c01 biowait+0x6c(60017fc1e80, 0, 183d400, 180c000, 790, 60017fc1e80) 02a1041f0cb1 ssd_send_scsi_cmd+0x394(760790, 2a1041f1668, 600018ef580, 1, 1, 0) 02a1041f0da1 ssd_send_scsi_TEST_UNIT_READY+0x100(600018ef580, 1, f2, 790, 0, 0) 02a1041f0eb1 ssd_get_media_info+0x64(760790, ffbfa8c8, 15, 600018ef580, 790, 8) 02a1041f0fc1 ssdioctl+0xb28(760790, 198b2800, 600018ef580, 0, 60006f20860, 2a1041f1adc) 02a1041f10d1 fop_ioctl+0x20(60007bae340, 42a, ffbfa8c8, 15, 60006f20860, 11ff9f0) 02a1041f1191 ioctl+0x184(3, 60007a14a88, ffbfa8c8, fff8, 73, 42a) 02a1041f12e1 syscall_trap32+0xcc(3, 42a, ffbfa8c8, fff8, 73, 70) stack pointer for thread 3fe0c80: 2a104208f11 [ 02a104208f11 cv_wait+0x38() ] 02a104208fc1 exitlwps+0x11c(0, 3fe0c80, 4a004002, 6001f3c8410, 20a0, 4a004002) 02a104209071 proc_exit+0x20(2, 2, 0, 6efc, 6000787b1a8, 1857400) 02a104209121 exit+8(2, 2, , 0, 6001f3c8410, 2) 02a1042091d1 post_syscall+0x3e8(4, 0, 3fe0e54, c9, 6000787b1a8, 1) 02a1042092e1 syscall_trap32+0x18c(0, 0, 0, 0, fed7bfa0, 4) This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Need Help: didn't create the pool as radiz but stripes
Boyd and all, Just an update of what happened and what the customer found out regarding the issue. === It does appear that the disk is fill up by 140G. I think I now know what happen. I created a raidz pool and I did not write any data to it before I just pulled out a disk. So I believe the zfs filesystem did not initialize yet. So this is why my zfs filesystem was unusable. Can you confirm this? But when I created a zfs filesystem and wrote data to it, it could now lose a disk and just be degraded. I tested this part by removing the disk partition in format. I will try this same test to re-duplicate my issue, but can you confirm for me if my zfs filesystem as a raidz requires me to write data to it first before it's really ready? [EMAIL PROTECTED] df -k Filesystemkbytesused avail capacity Mounted on /dev/dsk/c1t0d0s04136995 2918711 117691572%/ /devices 0 0 0 0%/devices ctfs 0 0 0 0%/system/contract proc 0 0 0 0%/proc mnttab 0 0 0 0%/etc/mnttab swap 5563996 616 5563380 1%/etc/svc/volatile objfs 0 0 0 0%/system/object /usr/lib/libc/libc_hwcap2.so.1 4136995 2918711 117691572%/lib/libc.so.1 fd 0 0 0 0%/dev/fd /dev/dsk/c1t0d0s54136995 78182 4017444 2%/var /dev/dsk/c1t0d0s741369954126 4091500 1%/tmp swap 5563400 20 5563380 1%/var/run /dev/dsk/c1t0d0s64136995 38674 4056952 1%/opt pool 210567315 210566773 0 100%/pool / [EMAIL PROTECTED] cd /pool /pool [EMAIL PROTECTED] ls -la total 421133452 drwxr-xr-x 2 root sys3 Aug 23 17:19 . drwxr-xr-x 25 root root 512 Aug 23 20:34 .. -rw--- 1 root root 171798691840 Aug 23 17:43 nullfile /pool [EMAIL PROTECTED] [EMAIL PROTECTED] zpool status pool: pool state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: none requested config: NAMESTATE READ WRITE CKSUM poolDEGRADED 0 0 0 raidz DEGRADED 0 0 0 c1t2d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c1t4d0 UNAVAIL 15.12 10.27 0 cannot open errors: No known data errors AND SECOND EMAIL: I'm unable to re-duplicate my failed zfs pool using raidz. As for the disk size bug (6288488 and 2140116), I have a few questions. The developer said that it would be fixed in u3. When is u3 suppose to be release? U2 just came out. Also, can or will = Any ideas when the Solaris 10 update 3 (11/06) be released? And would this be fixed in Solaris 10 update 2 (6/06)? Thanks to all of you. Arlina- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Need Help: didn't create the pool as radiz but stripes
On Thu, Aug 24, 2006 at 10:12:12AM -0600, Arlina Goce-Capiral wrote: > It does appear that the disk is fill up by 140G. So this confirms what I was saying, that they are only able to write ndisks-1 worth of data (in this case, ~68GB * (3-1) == ~136GB. So there is no unexpected behavior with respect to the size of their raid-z pool, just the known (and now fixed) bug. > I think I now know what happen. I created a raidz pool and I did not > write any data to it before I just pulled out a disk. So I believe the > zfs filesystem did not initialize yet. So this is why my zfs filesystem > was unusable. Can you confirm this? No, that should not be the case. As soon as the 'zfs' or 'zpool' command completes, everything will be on disk for the requested action. > But when I created a zfs filesystem and wrote data to it, it could now > lose a disk and just be degraded. I tested this part by removing the > disk partition in format. Well, it sounds like you are testing two different things: first you tried physically pulling out a disk, then you tried re-partitioning a disk. It sounds like there was a problem when you pulled out the disk. If you can describe the problem further (Did the machine panic? What was the panic message?) then perhaps we can diagnose it. > I will try this same test to re-duplicate my issue, but can you confirm > for me if my zfs filesystem as a raidz requires me to write data to it > first before it's really ready? No, that should not be the case. > Any ideas when the Solaris 10 update 3 (11/06) be released? I'm not sure, but November or December sounds about right. And of course, if they want the fix sooner they can always use Solaris Express or OpenSolaris! --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] unaccounted for daily growth in ZFS disk space usage
On Thu, Aug 24, 2006 at 07:07:45AM -0700, Joe Little wrote: > We finally flipped the switch on one of our ZFS-based servers, with > approximately 1TB of 2.8TB (3 stripes of 950MB or so, each of which is > a RAID5 volume on the adaptec card). We have snapshots every 4 hours > for the first few days. If you add up the snapshot references it > appears somewhat high versus daily use (mostly mail boxes, spam, etc > changing), but say an aggregate of no more than 400+MB a day. > > However, zfs list shows our daily pool as a whole, and per day we are > growing by .01TB, or more specifically 80GB a day. That's a far cry > different than the 400MB we can account for. Is it possible that > metadata/ditto blocks, or the like is trully growing that rapidly. By > our calculations, we will triple our disk space (sitting still) in 6 > months and use up the remaining 1.7TB. Of course, this is only with > 2-3 days of churn, but its an alarming rate where before on the NetApp > we didn't see anything close to this rate. How are you calculating this 400MB/day figure? Keep in mind that space "used" by each snapshot is the amount of space unique to that snapshot. Adding up the space "used" by all your snapshots is *not* the amount of space that they are all taking up cumulatively. For leaf filesystems (those with no descendents), you can calculate the space used by all snapshots as (fs's "used" - fs's "referenced"). How many filesystems do you have? Can you send me the output of 'zfs list' and 'zfs get -r all '? How much space did you expect to be using, and what data is that based on? Are you sure you aren't writing 80GB/day to your pool? --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: ZFS web admin - No items found.
Bill wrote: > Hi, experts, > > I install Solaris 10 06/06 x86 on vmware 5.5, and admin zfs by > command line and web, all is good. Web admin is more convenient, I > needn't type commands. But after my computer lost power , and > restarted, I get a problem on zfs web admin > (https://hostname:6789/zfs). > > The problem is, when I try to create a new storage pool from web , > it always shows "No items found" , but in fact there are 10 > harddisks available. > > I still can use zpool/zfs command line to create new pool, file > system, volumes. the command way works quickly and correctly. > > I have tried to restart the service (smcwebserver), no use. > > Anyone have the experience on it, is it a bug? ZFS may believe the disks are in use. What is the output of (as root) "/usr/lib/zfs/availdevs -d"? Thanks, Steve pgp8QDdCfubjZ.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS and very large directories
ZFS actually uses the ZAP to handle directory lookups. The ZAP is not a btree but a specialized hash table where a hash for each directory entry is generated based on that entry's name. Hence you won't be doing any sort of linear search through the entire directory for a file, a hash is generated from the file name and a lookup of that hash in the zap will be done. This is nice and speedy, even with 100,000 files in a directory. Noel On Aug 24, 2006, at 8:02 AM, Patrick Narkinsky wrote: Due to legacy constraints, I have a rather complicated system that is currently using Sun QFS (actually the SAM portion of it.) For a lot of reasons, I'd like to look at moving to ZFS, but would like a "sanity check" to make sure ZFS is suitable to this application. First of all, we are NOT using the cluster capabilities of SAMFS. Instead, we're using it as a way of dealing with one directory that contains approximately 100,000 entries. The question is this: I know from the specs that ZFS can handle a directory with this many entries, but what I'm actually wondering is how directory lookups are handled? That is, if I do a "cd foo05" in a directory with foo01 through foo10, will the filesystem have to scroll through all the directory contents to find foo05, or does it use a btree or something to handle this? This directory is, in turn, shared out over NFS. Are there any issues I should be aware of with this sort of installation? Thanks for any advice or input! Patrick Narkinsky Sr. System Engineer EDS This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS and very large directories
On Thu, Aug 24, 2006 at 10:46:27AM -0700, Noel Dellofano wrote: > ZFS actually uses the ZAP to handle directory lookups. The ZAP is > not a btree but a specialized hash table where a hash for each > directory entry is generated based on that entry's name. Hence you > won't be doing any sort of linear search through the entire directory > for a file, a hash is generated from the file name and a lookup of > that hash in the zap will be done. This is nice and speedy, even > with 100,000 files in a directory. I just tried creating 150,000 directories in a ZFS roto directory. It was speedy. Listing individual directories (lookup) is fast. Listing the large directory isn't, but that turns out to be either terminal I/O or collation in a UTF-8 locale (which is what I use; a simple DTrace script showed that to be my problem): % ptime ls ... real9.850 user6.263 <- ouch, UTF-8 hurts sys 0.245 % % LC_ALL=C ptime ls ... real4.112 <- terminal I/O hurts -- I should redirect to /dev/null user0.682 sys 0.252 % % LC_ALL=C ptime ls > /dev/null real0.793 user0.608 sys 0.184 <- not bad % % LC_ALL=C ptime sh -c 'echo *' ... real1.357 <- more compact output than ls(1) user0.970 sys 0.090 % % ptime ls -a x12345 . .. real0.022 user0.000 sys 0.002 % Awesome. Nico -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS and very large directories
On Thu, Aug 24, 2006 at 01:15:51PM -0500, Nicolas Williams wrote: > I just tried creating 150,000 directories in a ZFS roto directory. It > was speedy. Listing individual directories (lookup) is fast. Glad to hear that it's working well for you! > Listing the large directory isn't, but that turns out to be either > terminal I/O or collation in a UTF-8 locale (which is what I use; a > simple DTrace script showed that to be my problem): > > % ptime ls > ... > > real9.850 > user6.263 <- ouch, UTF-8 hurts > sys 0.245 Yep, beware of using 'ls' on large directories! See also: 6299769 'ls' memory usage is excessive 6299767 'ls -f' should not buffer output --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re[2]: [zfs-discuss] zpool hangs
Hello George, Thursday, August 24, 2006, 5:48:08 PM, you wrote: GW> Robert, GW> One of your disks is not responding. I've been trying to track down why GW> the scsi command is not being timed out but for now check out each of GW> the devices to make sure they are healthy. I know - I unmaped LUNs on the array. But it should time out. GW> BTW, if you capture a corefile let me know. Ooopss. already restarted -- Best regards, Robertmailto:[EMAIL PROTECTED] http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool hangs
Robert Milkowski wrote: Hello George, Thursday, August 24, 2006, 5:48:08 PM, you wrote: GW> Robert, GW> One of your disks is not responding. I've been trying to track down why GW> the scsi command is not being timed out but for now check out each of GW> the devices to make sure they are healthy. I know - I unmaped LUNs on the array. But it should time out. Agreed! I'm trying to track down what changes in the sd driver maybe contributing to this. - George GW> BTW, if you capture a corefile let me know. Ooopss. already restarted ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: Re: [zfs-discuss] unaccounted for daily growth in ZFS disk space usage
On 8/24/06, Matthew Ahrens <[EMAIL PROTECTED]> wrote: On Thu, Aug 24, 2006 at 07:07:45AM -0700, Joe Little wrote: > We finally flipped the switch on one of our ZFS-based servers, with > approximately 1TB of 2.8TB (3 stripes of 950MB or so, each of which is > a RAID5 volume on the adaptec card). We have snapshots every 4 hours > for the first few days. If you add up the snapshot references it > appears somewhat high versus daily use (mostly mail boxes, spam, etc > changing), but say an aggregate of no more than 400+MB a day. > > However, zfs list shows our daily pool as a whole, and per day we are > growing by .01TB, or more specifically 80GB a day. That's a far cry > different than the 400MB we can account for. Is it possible that > metadata/ditto blocks, or the like is trully growing that rapidly. By > our calculations, we will triple our disk space (sitting still) in 6 > months and use up the remaining 1.7TB. Of course, this is only with > 2-3 days of churn, but its an alarming rate where before on the NetApp > we didn't see anything close to this rate. How are you calculating this 400MB/day figure? Keep in mind that space "used" by each snapshot is the amount of space unique to that snapshot. Adding up the space "used" by all your snapshots is *not* the amount of space that they are all taking up cumulatively. For leaf filesystems (those with no descendents), you can calculate the space used by all snapshots as (fs's "used" - fs's "referenced"). How many filesystems do you have? Can you send me the output of 'zfs list' and 'zfs get -r all '? How much space did you expect to be using, and what data is that based on? Are you sure you aren't writing 80GB/day to your pool? --matt well, by deleting my 4-hourlies I reclaimed most of the data. To answer some of the questions, its about 15 filesystems (decendents included). I'm aware of the space used by snapshots overlapping. I was looking at the total space (zpool iostat reports) and seeing the diff per day. The 400MB/day was be inspection and by looking at our nominal growth on a netapp. It would appear that if one days many snapshots, there is an initial quick growth in disk usage, but once those snapshot meet their retention level (say 12), the growth would appear to match our typical 400MB/day. Time will prove this one way or other. By simply getting rid of hourly snapshots and collapsing to dailies for two days worth, I reverted to only ~1-2GB total growth, which is much more in line with expectations. For various reasons, I can't post the zfs list type results as yet. I'll need to get the ok for that first.. Sorry.. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: Re: [zfs-discuss] unaccounted for daily growth in ZFS disk space usage
On Thu, Aug 24, 2006 at 02:21:33PM -0700, Joe Little wrote: > well, by deleting my 4-hourlies I reclaimed most of the data. To > answer some of the questions, its about 15 filesystems (decendents > included). I'm aware of the space used by snapshots overlapping. I was > looking at the total space (zpool iostat reports) and seeing the diff > per day. The 400MB/day was be inspection and by looking at our nominal > growth on a netapp. > > It would appear that if one days many snapshots, there is an initial > quick growth in disk usage, but once those snapshot meet their > retention level (say 12), the growth would appear to match our typical > 400MB/day. Time will prove this one way or other. By simply getting > rid of hourly snapshots and collapsing to dailies for two days worth, > I reverted to only ~1-2GB total growth, which is much more in line > with expectations. OK, so sounds like there is no problem here, right? You were taking snapshots every 4 hours, which took up no more space than was needed, but more than you would like (and more than daily snapshots). Using daily snapshots the space usage is in line with daily snapshots on NetApp. > For various reasons, I can't post the zfs list type results as yet. > I'll need to get the ok for that first.. Sorry.. It sounds like there is no problem here so no need to post the output. --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: ZFS web admin - No items found.
When I run the command, it prompts: # /usr/lib/zfs/availdevs -d Segmentation Fault - core dumped. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] does anybody port the zfs webadmin to webmin?
Webmin is faster and light weight compared to SMC. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] does zfs can be failover between two mechine?
like two host share the storage, does in can failover ? does zfs can be stop and restart in cli? This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] does zfs can be failover between two mechine?
On Thu, 24 Aug 2006, Hawk Tsai wrote: does zfs can be failover between two mechine? like two host share the storage, does in can failover ? What do you mean exactly by "two host share the storage"? Are you looking for a distributed filesystem? ZFS is a local filesystem. Are you thinking of disks on a SAN which can be accessed by either machine? In that case, though only one machine can access the filesystem at one time, you could perhaps use "zpool import" to access the filesystem from the second machine if the first machine were to die. does zfs can be stop and restart in cli? Yes, see the documentation for the "zpool import" and "zpool export" commands (man zpool). Regards, Luke [caveat: I am only a ZFS user and not an expert!] ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: Re: Re: [zfs-discuss] unaccounted for daily growth in ZFS disk space usage
On 8/24/06, Matthew Ahrens <[EMAIL PROTECTED]> wrote: On Thu, Aug 24, 2006 at 02:21:33PM -0700, Joe Little wrote: > well, by deleting my 4-hourlies I reclaimed most of the data. To > answer some of the questions, its about 15 filesystems (decendents > included). I'm aware of the space used by snapshots overlapping. I was > looking at the total space (zpool iostat reports) and seeing the diff > per day. The 400MB/day was be inspection and by looking at our nominal > growth on a netapp. > > It would appear that if one days many snapshots, there is an initial > quick growth in disk usage, but once those snapshot meet their > retention level (say 12), the growth would appear to match our typical > 400MB/day. Time will prove this one way or other. By simply getting > rid of hourly snapshots and collapsing to dailies for two days worth, > I reverted to only ~1-2GB total growth, which is much more in line > with expectations. OK, so sounds like there is no problem here, right? You were taking snapshots every 4 hours, which took up no more space than was needed, but more than you would like (and more than daily snapshots). Using daily snapshots the space usage is in line with daily snapshots on NetApp. > For various reasons, I can't post the zfs list type results as yet. > I'll need to get the ok for that first.. Sorry.. It sounds like there is no problem here so no need to post the output. --matt Hi. The NetApp had the same snapshot schedule and didn't show the same growth in storage behind the base use. Again, I do suspect that ZFS snapshots (or perhaps the metadata) may be somewhat fatter over all than one would expect. Perhaps its the difference of default block sizing. It was an alarming rate until I suspected it was all in the snapshot overhead and reduced the overall number. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
