Re: [zfs-discuss] Checksum errors in storage pool
In the meantime, the SUN supporter did figure out that zdb does not work because zdb uses the information from /etc/zfs/zpool.cache. However, I did use zpool -R to import the pool, which did not update /etc/zfs/zpool.cache. Is there another method to map a dataset number to a filesystem? Hans Schnitzer H.-J. Schnitzer wrote: Hi, I am using ZFS under Solaris 10u3. After the defect of a 3510 Raid controller, I have several storage pools with defect objects. zpool status -xv prints a long list: DATASET OBJECT RANGE 4c0c 5dd lvl=0 blkid=2 28 b346lvl=0 blkid=9 3b31 15d lvl=0 blkid=1 3b31 15d lvl=0 blkid=2 3b31 15d lvl=0 blkid=2727 3b31 190 lvl=0 blkid=0 ... I know that the number in the column OBJECT identifies the inode number of the affected file. However, I have more than 1000 filesystems in each of the affected storage pools. So how do I identify the correct filesystem? According to http://blogs.sun.com/erickustarz/entry/damaged_files_and_zpool_status I have to use zdb. But I can't figure out how to use it. Can you help? Hans Schnitzer This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] C'mon ARC, stay small...
FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max: arc::print -tad { . . . c02e29e8 uint64_t size = 0t10527883264 c02e29f0 uint64_t p = 0t16381819904 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . Perhaps c_max does not do what I think it does? Thanks, /jim Jim Mauro wrote: Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06 (update 3). All file IO is mmap(file), read memory segment, unmap, close. Tweaked the arc size down via mdb to 1GB. I used that value because c_min was also 1GB, and I was not sure if c_max could be larger than c_minAnyway, I set c_max to 1GB. After a workload run: arc::print -tad { . . . c02e29e8 uint64_t size = 0t3099832832 c02e29f0 uint64_t p = 0t16540761088 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . size is at 3GB, with c_max at 1GB. What gives? I'm looking at the code now, but was under the impression c_max would limit ARC growth. Granted, it's not a factor of 10, and it's certainly much better than the out-of-the-box growth to 24GB (this is a 32GB x4500), so clearly ARC growth is being limited, but it still grew to 3X c_max. Thanks, /jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] C'mon ARC, stay small...
This seems a bit strange. What's the workload, and also, what's the output for: ARC_mru::print size lsize ARC_mfu::print size lsize and ARC_anon::print size For obvious reasons, the ARC can't evict buffers that are in use. Buffers that are available to be evicted should be on the mru or mfu list, so this output should be instructive. -j On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote: FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max: arc::print -tad { . . . c02e29e8 uint64_t size = 0t10527883264 c02e29f0 uint64_t p = 0t16381819904 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . Perhaps c_max does not do what I think it does? Thanks, /jim Jim Mauro wrote: Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06 (update 3). All file IO is mmap(file), read memory segment, unmap, close. Tweaked the arc size down via mdb to 1GB. I used that value because c_min was also 1GB, and I was not sure if c_max could be larger than c_minAnyway, I set c_max to 1GB. After a workload run: arc::print -tad { . . . c02e29e8 uint64_t size = 0t3099832832 c02e29f0 uint64_t p = 0t16540761088 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . size is at 3GB, with c_max at 1GB. What gives? I'm looking at the code now, but was under the impression c_max would limit ARC growth. Granted, it's not a factor of 10, and it's certainly much better than the out-of-the-box growth to 24GB (this is a 32GB x4500), so clearly ARC growth is being limited, but it still grew to 3X c_max. Thanks, /jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Recommended setup?
I don't Solaris dom0 does Pacifica (amd-v) yet. That would rule out windows for now. You can run centOS zones on SXCR. That just leaves freebsd (which hasn't got fantastic xen support either, despite Kip Macys excellent work). Unless you've got an app that needs that, zones sound like a much saner bet to me. On 13/03/07, Malachi de Ælfweald [EMAIL PROTECTED] wrote: I had thought about it, but from what I understand that limits the other VMs to Solaris. I have a few different administrators that are going to be running their own OSes (freebsd, linux, possibly windows), as well as some development ones (like jnode). From what I was able to find, that means that I need to run Xen with the newer AMD-V featureset; thus the reason for the new board and cpus. -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] C'mon ARC, stay small...
Gar. This isn't what I was hoping to see. Buffers that aren't available for eviction aren't listed in the lsize count. It looks like the MRU has grown to 10Gb and most of this could be successfully evicted. The calculation for determining if we evict from the MRU is in arc_adjust() and looks something like: top_sz = ARC_anon.size + ARC_mru.size Then if top_sz arc.p and ARC_mru.lsize 0 we evict the smaller of ARC_mru.lsize and top_size - arc.p In your previous message it looks like arc.p is (ARC_mru.size + ARC_anon.size). It might make sense to double-check these numbers together, so when you check the size and lsize again, also check arc.p. How/when did you configure arc_c_max? arc.p is supposed to be initialized to half of arc.c. Also, I assume that there's a reliable test case for reproducing this problem? Thanks, -j On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote: ARC_mru::print -d size lsize size = 0t10224433152 lsize = 0t10218960896 ARC_mfu::print -d size lsize size = 0t303450112 lsize = 0t289998848 ARC_anon::print -d size size = 0 So it looks like the MRU is running at 10GB... What does this tell us? Thanks, /jim [EMAIL PROTECTED] wrote: This seems a bit strange. What's the workload, and also, what's the output for: ARC_mru::print size lsize ARC_mfu::print size lsize and ARC_anon::print size For obvious reasons, the ARC can't evict buffers that are in use. Buffers that are available to be evicted should be on the mru or mfu list, so this output should be instructive. -j On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote: FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max: arc::print -tad { . . . c02e29e8 uint64_t size = 0t10527883264 c02e29f0 uint64_t p = 0t16381819904 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . Perhaps c_max does not do what I think it does? Thanks, /jim Jim Mauro wrote: Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06 (update 3). All file IO is mmap(file), read memory segment, unmap, close. Tweaked the arc size down via mdb to 1GB. I used that value because c_min was also 1GB, and I was not sure if c_max could be larger than c_minAnyway, I set c_max to 1GB. After a workload run: arc::print -tad { . . . c02e29e8 uint64_t size = 0t3099832832 c02e29f0 uint64_t p = 0t16540761088 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . size is at 3GB, with c_max at 1GB. What gives? I'm looking at the code now, but was under the impression c_max would limit ARC growth. Granted, it's not a factor of 10, and it's certainly much better than the out-of-the-box growth to 24GB (this is a 32GB x4500), so clearly ARC growth is being limited, but it still grew to 3X c_max. Thanks, /jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] C'mon ARC, stay small...
Something else to consider, depending upon how you set arc_c_max, you may just want to set arc_c and arc_p at the same time. If you try setting arc_c_max, and then setting arc_c to arc_c_max, and then set arc_p to arc_c / 2, do you still get this problem? -j On Thu, Mar 15, 2007 at 05:18:12PM -0700, [EMAIL PROTECTED] wrote: Gar. This isn't what I was hoping to see. Buffers that aren't available for eviction aren't listed in the lsize count. It looks like the MRU has grown to 10Gb and most of this could be successfully evicted. The calculation for determining if we evict from the MRU is in arc_adjust() and looks something like: top_sz = ARC_anon.size + ARC_mru.size Then if top_sz arc.p and ARC_mru.lsize 0 we evict the smaller of ARC_mru.lsize and top_size - arc.p In your previous message it looks like arc.p is (ARC_mru.size + ARC_anon.size). It might make sense to double-check these numbers together, so when you check the size and lsize again, also check arc.p. How/when did you configure arc_c_max? arc.p is supposed to be initialized to half of arc.c. Also, I assume that there's a reliable test case for reproducing this problem? Thanks, -j On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote: ARC_mru::print -d size lsize size = 0t10224433152 lsize = 0t10218960896 ARC_mfu::print -d size lsize size = 0t303450112 lsize = 0t289998848 ARC_anon::print -d size size = 0 So it looks like the MRU is running at 10GB... What does this tell us? Thanks, /jim [EMAIL PROTECTED] wrote: This seems a bit strange. What's the workload, and also, what's the output for: ARC_mru::print size lsize ARC_mfu::print size lsize and ARC_anon::print size For obvious reasons, the ARC can't evict buffers that are in use. Buffers that are available to be evicted should be on the mru or mfu list, so this output should be instructive. -j On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote: FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max: arc::print -tad { . . . c02e29e8 uint64_t size = 0t10527883264 c02e29f0 uint64_t p = 0t16381819904 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . Perhaps c_max does not do what I think it does? Thanks, /jim Jim Mauro wrote: Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06 (update 3). All file IO is mmap(file), read memory segment, unmap, close. Tweaked the arc size down via mdb to 1GB. I used that value because c_min was also 1GB, and I was not sure if c_max could be larger than c_minAnyway, I set c_max to 1GB. After a workload run: arc::print -tad { . . . c02e29e8 uint64_t size = 0t3099832832 c02e29f0 uint64_t p = 0t16540761088 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . size is at 3GB, with c_max at 1GB. What gives? I'm looking at the code now, but was under the impression c_max would limit ARC growth. Granted, it's not a factor of 10, and it's certainly much better than the out-of-the-box growth to 24GB (this is a 32GB x4500), so clearly ARC growth is being limited, but it still grew to 3X c_max. Thanks, /jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] C'mon ARC, stay small...
How/when did you configure arc_c_max? Immediately following a reboot, I set arc.c_max using mdb, then verified reading the arc structure again. arc.p is supposed to be initialized to half of arc.c. Also, I assume that there's a reliable test case for reproducing this problem? Yep. I'm using a x4500 in-house to sort out performance of a customer test case that uses mmap. We acquired the new DIMMs to bring the x4500 to 32GB, since the workload has a 64GB working set size, and we were clobbering a 16GB thumper. We wanted to see how doubling memory may help. I'm trying clamp the ARC size because for mmap-intensive workloads, it seems to hurt more than help (although, based on experiments up to this point, it's not hurting a lot). I'll do another reboot, and run it all down for you serially... /jim Thanks, -j On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote: ARC_mru::print -d size lsize size = 0t10224433152 lsize = 0t10218960896 ARC_mfu::print -d size lsize size = 0t303450112 lsize = 0t289998848 ARC_anon::print -d size size = 0 So it looks like the MRU is running at 10GB... What does this tell us? Thanks, /jim [EMAIL PROTECTED] wrote: This seems a bit strange. What's the workload, and also, what's the output for: ARC_mru::print size lsize ARC_mfu::print size lsize and ARC_anon::print size For obvious reasons, the ARC can't evict buffers that are in use. Buffers that are available to be evicted should be on the mru or mfu list, so this output should be instructive. -j On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote: FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max: arc::print -tad { . . . c02e29e8 uint64_t size = 0t10527883264 c02e29f0 uint64_t p = 0t16381819904 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . Perhaps c_max does not do what I think it does? Thanks, /jim Jim Mauro wrote: Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06 (update 3). All file IO is mmap(file), read memory segment, unmap, close. Tweaked the arc size down via mdb to 1GB. I used that value because c_min was also 1GB, and I was not sure if c_max could be larger than c_minAnyway, I set c_max to 1GB. After a workload run: arc::print -tad { . . . c02e29e8 uint64_t size = 0t3099832832 c02e29f0 uint64_t p = 0t16540761088 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . size is at 3GB, with c_max at 1GB. What gives? I'm looking at the code now, but was under the impression c_max would limit ARC growth. Granted, it's not a factor of 10, and it's certainly much better than the out-of-the-box growth to 24GB (this is a 32GB x4500), so clearly ARC growth is being limited, but it still grew to 3X c_max. Thanks, /jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] C'mon ARC, stay small...
I suppose I should have been more forward about making my last point. If the arc_c_max isn't set in /etc/system, I don't believe that the ARC will initialize arc.p to the correct value. I could be wrong about this; however, next time you set c_max, set c to the same value as c_max and set p to half of c. Let me know if this addresses the problem or not. -j How/when did you configure arc_c_max? Immediately following a reboot, I set arc.c_max using mdb, then verified reading the arc structure again. arc.p is supposed to be initialized to half of arc.c. Also, I assume that there's a reliable test case for reproducing this problem? Yep. I'm using a x4500 in-house to sort out performance of a customer test case that uses mmap. We acquired the new DIMMs to bring the x4500 to 32GB, since the workload has a 64GB working set size, and we were clobbering a 16GB thumper. We wanted to see how doubling memory may help. I'm trying clamp the ARC size because for mmap-intensive workloads, it seems to hurt more than help (although, based on experiments up to this point, it's not hurting a lot). I'll do another reboot, and run it all down for you serially... /jim Thanks, -j On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote: ARC_mru::print -d size lsize size = 0t10224433152 lsize = 0t10218960896 ARC_mfu::print -d size lsize size = 0t303450112 lsize = 0t289998848 ARC_anon::print -d size size = 0 So it looks like the MRU is running at 10GB... What does this tell us? Thanks, /jim [EMAIL PROTECTED] wrote: This seems a bit strange. What's the workload, and also, what's the output for: ARC_mru::print size lsize ARC_mfu::print size lsize and ARC_anon::print size For obvious reasons, the ARC can't evict buffers that are in use. Buffers that are available to be evicted should be on the mru or mfu list, so this output should be instructive. -j On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote: FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max: arc::print -tad { . . . c02e29e8 uint64_t size = 0t10527883264 c02e29f0 uint64_t p = 0t16381819904 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . Perhaps c_max does not do what I think it does? Thanks, /jim Jim Mauro wrote: Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06 (update 3). All file IO is mmap(file), read memory segment, unmap, close. Tweaked the arc size down via mdb to 1GB. I used that value because c_min was also 1GB, and I was not sure if c_max could be larger than c_minAnyway, I set c_max to 1GB. After a workload run: arc::print -tad { . . . c02e29e8 uint64_t size = 0t3099832832 c02e29f0 uint64_t p = 0t16540761088 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . size is at 3GB, with c_max at 1GB. What gives? I'm looking at the code now, but was under the impression c_max would limit ARC growth. Granted, it's not a factor of 10, and it's certainly much better than the out-of-the-box growth to 24GB (this is a 32GB x4500), so clearly ARC growth is being limited, but it still grew to 3X c_max. Thanks, /jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] C'mon ARC, stay small...
Following a reboot: arc::print -tad { . . . c02e29e8 uint64_t size = 0t299008 c02e29f0 uint64_t p = 0t16588228608 c02e29f8 uint64_t c = 0t33176457216 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t33176457216 . . . } c02e2a08 /Z 0x2000 --- set c_max to 512MB arc+0x48: 0x7b9789000 = 0x2000 arc::print -tad { . . . c02e29e8 uint64_t size = 0t299008 c02e29f0 uint64_t p = 0t16588228608 c02e29f8 uint64_t c = 0t33176457216 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t536870912 - c_max is 512MB . . . } ARC_mru::print -d size lsize size = 0t294912 lsize = 0t32768 Run the workload a couple times... c02e29e8 uint64_t size = 0t27121205248 --- ARC size is 27GB c02e29f0 uint64_t p = 0t10551351442 c02e29f8 uint64_t c = 0t27121332576 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t536870912 - c_max is 512MB ARC_mru::print -d size lsize size = 0t223985664 lsize = 0t221839360 ARC_mfu::print -d size lsize size = 0t26897219584 -- MFU list is almost 27GB ... lsize = 0t26869121024 Thanks, /jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] C'mon ARC, stay small...
Will try that now... /jim [EMAIL PROTECTED] wrote: I suppose I should have been more forward about making my last point. If the arc_c_max isn't set in /etc/system, I don't believe that the ARC will initialize arc.p to the correct value. I could be wrong about this; however, next time you set c_max, set c to the same value as c_max and set p to half of c. Let me know if this addresses the problem or not. -j How/when did you configure arc_c_max? Immediately following a reboot, I set arc.c_max using mdb, then verified reading the arc structure again. arc.p is supposed to be initialized to half of arc.c. Also, I assume that there's a reliable test case for reproducing this problem? Yep. I'm using a x4500 in-house to sort out performance of a customer test case that uses mmap. We acquired the new DIMMs to bring the x4500 to 32GB, since the workload has a 64GB working set size, and we were clobbering a 16GB thumper. We wanted to see how doubling memory may help. I'm trying clamp the ARC size because for mmap-intensive workloads, it seems to hurt more than help (although, based on experiments up to this point, it's not hurting a lot). I'll do another reboot, and run it all down for you serially... /jim Thanks, -j On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote: ARC_mru::print -d size lsize size = 0t10224433152 lsize = 0t10218960896 ARC_mfu::print -d size lsize size = 0t303450112 lsize = 0t289998848 ARC_anon::print -d size size = 0 So it looks like the MRU is running at 10GB... What does this tell us? Thanks, /jim [EMAIL PROTECTED] wrote: This seems a bit strange. What's the workload, and also, what's the output for: ARC_mru::print size lsize ARC_mfu::print size lsize and ARC_anon::print size For obvious reasons, the ARC can't evict buffers that are in use. Buffers that are available to be evicted should be on the mru or mfu list, so this output should be instructive. -j On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote: FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max: arc::print -tad { . . . c02e29e8 uint64_t size = 0t10527883264 c02e29f0 uint64_t p = 0t16381819904 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . Perhaps c_max does not do what I think it does? Thanks, /jim Jim Mauro wrote: Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06 (update 3). All file IO is mmap(file), read memory segment, unmap, close. Tweaked the arc size down via mdb to 1GB. I used that value because c_min was also 1GB, and I was not sure if c_max could be larger than c_minAnyway, I set c_max to 1GB. After a workload run: arc::print -tad { . . . c02e29e8 uint64_t size = 0t3099832832 c02e29f0 uint64_t p = 0t16540761088 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . size is at 3GB, with c_max at 1GB. What gives? I'm looking at the code now, but was under the impression c_max would limit ARC growth. Granted, it's not a factor of 10, and it's certainly much better than the out-of-the-box growth to 24GB (this is a 32GB x4500), so clearly ARC growth is being limited, but it still grew to 3X c_max. Thanks, /jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] C'mon ARC, stay small...
All righty...I set c_max to 512MB, c to 512MB, and p to 256MB... arc::print -tad { ... c02e29e8 uint64_t size = 0t299008 c02e29f0 uint64_t p = 0t16588228608 c02e29f8 uint64_t c = 0t33176457216 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t33176457216 ... } c02e2a08 /Z 0x2000 arc+0x48: 0x7b9789000 = 0x2000 c02e29f8 /Z 0x2000 arc+0x38: 0x7b9789000 = 0x2000 c02e29f0 /Z 0x1000 arc+0x30: 0x3dcbc4800 = 0x1000 arc::print -tad { ... c02e29e8 uint64_t size = 0t299008 c02e29f0 uint64_t p = 0t268435456 -- p is 256MB c02e29f8 uint64_t c = 0t536870912 -- c is 512MB c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t536870912--- c_max is 512MB ... } After a few runs of the workload ... arc::print -d size size = 0t536788992 Ah - looks like we're out of the woods. The ARC remains clamped at 512MB. Thanks! /jim [EMAIL PROTECTED] wrote: I suppose I should have been more forward about making my last point. If the arc_c_max isn't set in /etc/system, I don't believe that the ARC will initialize arc.p to the correct value. I could be wrong about this; however, next time you set c_max, set c to the same value as c_max and set p to half of c. Let me know if this addresses the problem or not. -j How/when did you configure arc_c_max? Immediately following a reboot, I set arc.c_max using mdb, then verified reading the arc structure again. arc.p is supposed to be initialized to half of arc.c. Also, I assume that there's a reliable test case for reproducing this problem? Yep. I'm using a x4500 in-house to sort out performance of a customer test case that uses mmap. We acquired the new DIMMs to bring the x4500 to 32GB, since the workload has a 64GB working set size, and we were clobbering a 16GB thumper. We wanted to see how doubling memory may help. I'm trying clamp the ARC size because for mmap-intensive workloads, it seems to hurt more than help (although, based on experiments up to this point, it's not hurting a lot). I'll do another reboot, and run it all down for you serially... /jim Thanks, -j On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote: ARC_mru::print -d size lsize size = 0t10224433152 lsize = 0t10218960896 ARC_mfu::print -d size lsize size = 0t303450112 lsize = 0t289998848 ARC_anon::print -d size size = 0 So it looks like the MRU is running at 10GB... What does this tell us? Thanks, /jim [EMAIL PROTECTED] wrote: This seems a bit strange. What's the workload, and also, what's the output for: ARC_mru::print size lsize ARC_mfu::print size lsize and ARC_anon::print size For obvious reasons, the ARC can't evict buffers that are in use. Buffers that are available to be evicted should be on the mru or mfu list, so this output should be instructive. -j On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote: FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max: arc::print -tad { . . . c02e29e8 uint64_t size = 0t10527883264 c02e29f0 uint64_t p = 0t16381819904 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . Perhaps c_max does not do what I think it does? Thanks, /jim Jim Mauro wrote: Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06 (update 3). All file IO is mmap(file), read memory segment, unmap, close. Tweaked the arc size down via mdb to 1GB. I used that value because c_min was also 1GB, and I was not sure if c_max could be larger than c_minAnyway, I set c_max to 1GB. After a workload run: arc::print -tad { . . . c02e29e8 uint64_t size = 0t3099832832 c02e29f0 uint64_t p = 0t16540761088 c02e29f8 uint64_t c = 0t1070318720 c02e2a00 uint64_t c_min = 0t1070318720 c02e2a08 uint64_t c_max = 0t1070318720 . . . size is at 3GB, with c_max at 1GB. What gives? I'm looking at the code now, but was under the impression c_max would limit ARC growth. Granted, it's not a factor of 10, and it's certainly much better than the out-of-the-box growth to 24GB (this is a 32GB x4500), so clearly ARC growth is being limited, but it still grew to 3X c_max. Thanks, /jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org