Re: [zfs-discuss] Checksum errors in storage pool

2007-03-15 Thread Hans-Juergen Schnitzer


In the meantime, the SUN supporter did figure out that zdb does not work
because zdb uses the information from /etc/zfs/zpool.cache. However,
I did use zpool -R to import the pool, which did not update 
/etc/zfs/zpool.cache. Is there another method to map a dataset

number to a filesystem?

Hans Schnitzer

H.-J. Schnitzer wrote:

Hi,

I am using ZFS under Solaris 10u3.

After the defect of a 3510 Raid controller, I have several storage pools
with defect objects. zpool status -xv prints a long list:

  DATASET  OBJECT  RANGE
  4c0c 5dd lvl=0 blkid=2
  28   b346lvl=0 blkid=9
  3b31 15d lvl=0 blkid=1
  3b31 15d lvl=0 blkid=2
  3b31 15d lvl=0 blkid=2727
  3b31 190 lvl=0 blkid=0
  ...

I know that the number in the column OBJECT identifies the inode number
of the affected file. 
However, I have more than 1000 filesystems  in each of the 
affected storage pools. So how do I identify the correct filesystem?

According to 
http://blogs.sun.com/erickustarz/entry/damaged_files_and_zpool_status
I have to use zdb. But I can't figure out how to use it. Can you help?

Hans Schnitzer
 
 
This message posted from opensolaris.org

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss





___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] C'mon ARC, stay small...

2007-03-15 Thread Jim Mauro


FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max:


 arc::print -tad
{
. . .
   c02e29e8 uint64_t size = 0t10527883264
   c02e29f0 uint64_t p = 0t16381819904
   c02e29f8 uint64_t c = 0t1070318720
   c02e2a00 uint64_t c_min = 0t1070318720
   c02e2a08 uint64_t c_max = 0t1070318720
. . .

Perhaps c_max does not do what I think it does?

Thanks,
/jim


Jim Mauro wrote:

Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06
(update 3). All file IO is mmap(file), read memory segment, unmap, close.

Tweaked the arc size down via mdb to 1GB. I used that value because
c_min was also 1GB, and I was not sure if c_max could be larger than
c_minAnyway, I set c_max to 1GB.

After a workload run:
 arc::print -tad
{
. . .
  c02e29e8 uint64_t size = 0t3099832832
  c02e29f0 uint64_t p = 0t16540761088
  c02e29f8 uint64_t c = 0t1070318720
  c02e2a00 uint64_t c_min = 0t1070318720
  c02e2a08 uint64_t c_max = 0t1070318720
. . .

size is at 3GB, with c_max at 1GB.

What gives? I'm looking at the code now, but was under the impression
c_max would limit ARC growth. Granted, it's not a factor of 10, and
it's certainly much better than the out-of-the-box growth to 24GB
(this is a 32GB x4500), so clearly ARC growth is being limited, but it
still grew to 3X c_max.

Thanks,
/jim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] C'mon ARC, stay small...

2007-03-15 Thread johansen-osdev
This seems a bit strange.  What's the workload, and also, what's the
output for:

 ARC_mru::print size lsize
 ARC_mfu::print size lsize
and
 ARC_anon::print size

For obvious reasons, the ARC can't evict buffers that are in use.
Buffers that are available to be evicted should be on the mru or mfu
list, so this output should be instructive.

-j

On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote:
 
 FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max:
 
 
  arc::print -tad
 {
 . . .
c02e29e8 uint64_t size = 0t10527883264
c02e29f0 uint64_t p = 0t16381819904
c02e29f8 uint64_t c = 0t1070318720
c02e2a00 uint64_t c_min = 0t1070318720
c02e2a08 uint64_t c_max = 0t1070318720
 . . .
 
 Perhaps c_max does not do what I think it does?
 
 Thanks,
 /jim
 
 
 Jim Mauro wrote:
 Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06
 (update 3). All file IO is mmap(file), read memory segment, unmap, close.
 
 Tweaked the arc size down via mdb to 1GB. I used that value because
 c_min was also 1GB, and I was not sure if c_max could be larger than
 c_minAnyway, I set c_max to 1GB.
 
 After a workload run:
  arc::print -tad
 {
 . . .
   c02e29e8 uint64_t size = 0t3099832832
   c02e29f0 uint64_t p = 0t16540761088
   c02e29f8 uint64_t c = 0t1070318720
   c02e2a00 uint64_t c_min = 0t1070318720
   c02e2a08 uint64_t c_max = 0t1070318720
 . . .
 
 size is at 3GB, with c_max at 1GB.
 
 What gives? I'm looking at the code now, but was under the impression
 c_max would limit ARC growth. Granted, it's not a factor of 10, and
 it's certainly much better than the out-of-the-box growth to 24GB
 (this is a 32GB x4500), so clearly ARC growth is being limited, but it
 still grew to 3X c_max.
 
 Thanks,
 /jim
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Recommended setup?

2007-03-15 Thread Dick Davies

I don't Solaris dom0 does Pacifica (amd-v) yet.
That would rule out windows for now.

You can run centOS zones on SXCR.

That just leaves freebsd (which hasn't got fantastic xen support either,
despite Kip Macys excellent work).

Unless you've got an app that needs that, zones sound like a much saner bet
to me.

On 13/03/07, Malachi de Ælfweald [EMAIL PROTECTED] wrote:

I had thought about it, but from what I understand that limits the other VMs
to Solaris. I have a few different administrators that are going to be
running their own OSes (freebsd, linux, possibly windows), as well as some
development ones (like jnode).  From what I was able to find, that means
that I need to run Xen with the newer AMD-V featureset; thus the reason for
the new board and cpus.


--
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] C'mon ARC, stay small...

2007-03-15 Thread johansen-osdev
Gar.  This isn't what I was hoping to see.  Buffers that aren't
available for eviction aren't listed in the lsize count.  It looks like
the MRU has grown to 10Gb and most of this could be successfully
evicted.

The calculation for determining if we evict from the MRU is in
arc_adjust() and looks something like:

top_sz = ARC_anon.size + ARC_mru.size

Then if top_sz  arc.p and ARC_mru.lsize  0 we evict the smaller of
ARC_mru.lsize and top_size - arc.p

In your previous message it looks like arc.p is  (ARC_mru.size +
ARC_anon.size).  It might make sense to double-check these numbers
together, so when you check the size and lsize again, also check arc.p.

How/when did you configure arc_c_max?  arc.p is supposed to be
initialized to half of arc.c.  Also, I assume that there's a reliable
test case for reproducing this problem?

Thanks,

-j

On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote:
 
 
  ARC_mru::print -d size lsize
 size = 0t10224433152
 lsize = 0t10218960896
  ARC_mfu::print -d size lsize
 size = 0t303450112
 lsize = 0t289998848
  ARC_anon::print -d size
 size = 0
 
 
 So it looks like the MRU is running at 10GB...
 
 What does this tell us?
 
 Thanks,
 /jim
 
 
 
 [EMAIL PROTECTED] wrote:
 This seems a bit strange.  What's the workload, and also, what's the
 output for:
 
   
 ARC_mru::print size lsize
 ARC_mfu::print size lsize
 
 and
   
 ARC_anon::print size
 
 
 For obvious reasons, the ARC can't evict buffers that are in use.
 Buffers that are available to be evicted should be on the mru or mfu
 list, so this output should be instructive.
 
 -j
 
 On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote:
   
 FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max:
 
 
 
 arc::print -tad
   
 {
 . . .
c02e29e8 uint64_t size = 0t10527883264
c02e29f0 uint64_t p = 0t16381819904
c02e29f8 uint64_t c = 0t1070318720
c02e2a00 uint64_t c_min = 0t1070318720
c02e2a08 uint64_t c_max = 0t1070318720
 . . .
 
 Perhaps c_max does not do what I think it does?
 
 Thanks,
 /jim
 
 
 Jim Mauro wrote:
 
 Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06
 (update 3). All file IO is mmap(file), read memory segment, unmap, close.
 
 Tweaked the arc size down via mdb to 1GB. I used that value because
 c_min was also 1GB, and I was not sure if c_max could be larger than
 c_minAnyway, I set c_max to 1GB.
 
 After a workload run:
   
 arc::print -tad
 
 {
 . . .
  c02e29e8 uint64_t size = 0t3099832832
  c02e29f0 uint64_t p = 0t16540761088
  c02e29f8 uint64_t c = 0t1070318720
  c02e2a00 uint64_t c_min = 0t1070318720
  c02e2a08 uint64_t c_max = 0t1070318720
 . . .
 
 size is at 3GB, with c_max at 1GB.
 
 What gives? I'm looking at the code now, but was under the impression
 c_max would limit ARC growth. Granted, it's not a factor of 10, and
 it's certainly much better than the out-of-the-box growth to 24GB
 (this is a 32GB x4500), so clearly ARC growth is being limited, but it
 still grew to 3X c_max.
 
 Thanks,
 /jim
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
   
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
 
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] C'mon ARC, stay small...

2007-03-15 Thread johansen-osdev
Something else to consider, depending upon how you set arc_c_max, you
may just want to set arc_c and arc_p at the same time.  If you try
setting arc_c_max, and then setting arc_c to arc_c_max, and then set
arc_p to arc_c / 2, do you still get this problem?

-j

On Thu, Mar 15, 2007 at 05:18:12PM -0700, [EMAIL PROTECTED] wrote:
 Gar.  This isn't what I was hoping to see.  Buffers that aren't
 available for eviction aren't listed in the lsize count.  It looks like
 the MRU has grown to 10Gb and most of this could be successfully
 evicted.
 
 The calculation for determining if we evict from the MRU is in
 arc_adjust() and looks something like:
 
 top_sz = ARC_anon.size + ARC_mru.size
 
 Then if top_sz  arc.p and ARC_mru.lsize  0 we evict the smaller of
 ARC_mru.lsize and top_size - arc.p
 
 In your previous message it looks like arc.p is  (ARC_mru.size +
 ARC_anon.size).  It might make sense to double-check these numbers
 together, so when you check the size and lsize again, also check arc.p.
 
 How/when did you configure arc_c_max?  arc.p is supposed to be
 initialized to half of arc.c.  Also, I assume that there's a reliable
 test case for reproducing this problem?
 
 Thanks,
 
 -j
 
 On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote:
  
  
   ARC_mru::print -d size lsize
  size = 0t10224433152
  lsize = 0t10218960896
   ARC_mfu::print -d size lsize
  size = 0t303450112
  lsize = 0t289998848
   ARC_anon::print -d size
  size = 0
  
  
  So it looks like the MRU is running at 10GB...
  
  What does this tell us?
  
  Thanks,
  /jim
  
  
  
  [EMAIL PROTECTED] wrote:
  This seems a bit strange.  What's the workload, and also, what's the
  output for:
  

  ARC_mru::print size lsize
  ARC_mfu::print size lsize
  
  and

  ARC_anon::print size
  
  
  For obvious reasons, the ARC can't evict buffers that are in use.
  Buffers that are available to be evicted should be on the mru or mfu
  list, so this output should be instructive.
  
  -j
  
  On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote:

  FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max:
  
  
  
  arc::print -tad

  {
  . . .
 c02e29e8 uint64_t size = 0t10527883264
 c02e29f0 uint64_t p = 0t16381819904
 c02e29f8 uint64_t c = 0t1070318720
 c02e2a00 uint64_t c_min = 0t1070318720
 c02e2a08 uint64_t c_max = 0t1070318720
  . . .
  
  Perhaps c_max does not do what I think it does?
  
  Thanks,
  /jim
  
  
  Jim Mauro wrote:
  
  Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06
  (update 3). All file IO is mmap(file), read memory segment, unmap, close.
  
  Tweaked the arc size down via mdb to 1GB. I used that value because
  c_min was also 1GB, and I was not sure if c_max could be larger than
  c_minAnyway, I set c_max to 1GB.
  
  After a workload run:

  arc::print -tad
  
  {
  . . .
   c02e29e8 uint64_t size = 0t3099832832
   c02e29f0 uint64_t p = 0t16540761088
   c02e29f8 uint64_t c = 0t1070318720
   c02e2a00 uint64_t c_min = 0t1070318720
   c02e2a08 uint64_t c_max = 0t1070318720
  . . .
  
  size is at 3GB, with c_max at 1GB.
  
  What gives? I'm looking at the code now, but was under the impression
  c_max would limit ARC growth. Granted, it's not a factor of 10, and
  it's certainly much better than the out-of-the-box growth to 24GB
  (this is a 32GB x4500), so clearly ARC growth is being limited, but it
  still grew to 3X c_max.
  
  Thanks,
  /jim
  ___
  zfs-discuss mailing list
  zfs-discuss@opensolaris.org
  http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

  ___
  zfs-discuss mailing list
  zfs-discuss@opensolaris.org
  http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
  
  ___
  zfs-discuss mailing list
  zfs-discuss@opensolaris.org
  http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] C'mon ARC, stay small...

2007-03-15 Thread Jim Mauro




How/when did you configure arc_c_max?  

Immediately following a reboot, I set arc.c_max using mdb,
then verified reading the arc structure again.

arc.p is supposed to be
initialized to half of arc.c.  Also, I assume that there's a reliable
test case for reproducing this problem?
  

Yep. I'm using a x4500 in-house to sort out performance of a customer test
case that uses mmap. We acquired the new DIMMs to bring the
x4500 to 32GB, since the workload has a 64GB working set size,
and we were clobbering a 16GB thumper. We wanted to see how doubling
memory may help.

I'm trying clamp the ARC size because for mmap-intensive workloads,
it seems to hurt more than help (although, based on experiments up to this
point, it's not hurting a lot).

I'll do another reboot, and run it all down for you serially...

/jim


Thanks,

-j

On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote:
  


ARC_mru::print -d size lsize
  

size = 0t10224433152
lsize = 0t10218960896


ARC_mfu::print -d size lsize
  

size = 0t303450112
lsize = 0t289998848


ARC_anon::print -d size
  

size = 0

So it looks like the MRU is running at 10GB...


What does this tell us?

Thanks,
/jim



[EMAIL PROTECTED] wrote:


This seems a bit strange.  What's the workload, and also, what's the
output for:

 
  

ARC_mru::print size lsize
ARC_mfu::print size lsize
   


and
 
  

ARC_anon::print size
   


For obvious reasons, the ARC can't evict buffers that are in use.
Buffers that are available to be evicted should be on the mru or mfu
list, so this output should be instructive.

-j

On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote:
 
  

FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max:


   


arc::print -tad
 
  

{
. . .
  c02e29e8 uint64_t size = 0t10527883264
  c02e29f0 uint64_t p = 0t16381819904
  c02e29f8 uint64_t c = 0t1070318720
  c02e2a00 uint64_t c_min = 0t1070318720
  c02e2a08 uint64_t c_max = 0t1070318720
. . .

Perhaps c_max does not do what I think it does?

Thanks,
/jim


Jim Mauro wrote:
   


Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06
(update 3). All file IO is mmap(file), read memory segment, unmap, close.

Tweaked the arc size down via mdb to 1GB. I used that value because
c_min was also 1GB, and I was not sure if c_max could be larger than
c_minAnyway, I set c_max to 1GB.

After a workload run:
 
  

arc::print -tad
   


{
. . .
c02e29e8 uint64_t size = 0t3099832832
c02e29f0 uint64_t p = 0t16540761088
c02e29f8 uint64_t c = 0t1070318720
c02e2a00 uint64_t c_min = 0t1070318720
c02e2a08 uint64_t c_max = 0t1070318720
. . .

size is at 3GB, with c_max at 1GB.

What gives? I'm looking at the code now, but was under the impression
c_max would limit ARC growth. Granted, it's not a factor of 10, and
it's certainly much better than the out-of-the-box growth to 24GB
(this is a 32GB x4500), so clearly ARC growth is being limited, but it
still grew to 3X c_max.

Thanks,
/jim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
 
  

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
   


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] C'mon ARC, stay small...

2007-03-15 Thread johansen-osdev
I suppose I should have been more forward about making my last point.
If the arc_c_max isn't set in /etc/system, I don't believe that the ARC
will initialize arc.p to the correct value.   I could be wrong about
this; however, next time you set c_max, set c to the same value as c_max
and set p to half of c.  Let me know if this addresses the problem or
not.

-j

 How/when did you configure arc_c_max?  
 Immediately following a reboot, I set arc.c_max using mdb,
 then verified reading the arc structure again.
 arc.p is supposed to be
 initialized to half of arc.c.  Also, I assume that there's a reliable
 test case for reproducing this problem?
   
 Yep. I'm using a x4500 in-house to sort out performance of a customer test
 case that uses mmap. We acquired the new DIMMs to bring the
 x4500 to 32GB, since the workload has a 64GB working set size,
 and we were clobbering a 16GB thumper. We wanted to see how doubling
 memory may help.
 
 I'm trying clamp the ARC size because for mmap-intensive workloads,
 it seems to hurt more than help (although, based on experiments up to this
 point, it's not hurting a lot).
 
 I'll do another reboot, and run it all down for you serially...
 
 /jim
 
 Thanks,
 
 -j
 
 On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote:
   
 
 ARC_mru::print -d size lsize
   
 size = 0t10224433152
 lsize = 0t10218960896
 
 ARC_mfu::print -d size lsize
   
 size = 0t303450112
 lsize = 0t289998848
 
 ARC_anon::print -d size
   
 size = 0
 
 So it looks like the MRU is running at 10GB...
 
 What does this tell us?
 
 Thanks,
 /jim
 
 
 
 [EMAIL PROTECTED] wrote:
 
 This seems a bit strange.  What's the workload, and also, what's the
 output for:
 
  
   
 ARC_mru::print size lsize
 ARC_mfu::print size lsize

 
 and
  
   
 ARC_anon::print size

 
 For obvious reasons, the ARC can't evict buffers that are in use.
 Buffers that are available to be evicted should be on the mru or mfu
 list, so this output should be instructive.
 
 -j
 
 On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote:
  
   
 FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max:
 
 

 
 arc::print -tad
  
   
 {
 . . .
   c02e29e8 uint64_t size = 0t10527883264
   c02e29f0 uint64_t p = 0t16381819904
   c02e29f8 uint64_t c = 0t1070318720
   c02e2a00 uint64_t c_min = 0t1070318720
   c02e2a08 uint64_t c_max = 0t1070318720
 . . .
 
 Perhaps c_max does not do what I think it does?
 
 Thanks,
 /jim
 
 
 Jim Mauro wrote:

 
 Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06
 (update 3). All file IO is mmap(file), read memory segment, unmap, 
 close.
 
 Tweaked the arc size down via mdb to 1GB. I used that value because
 c_min was also 1GB, and I was not sure if c_max could be larger than
 c_minAnyway, I set c_max to 1GB.
 
 After a workload run:
  
   
 arc::print -tad

 
 {
 . . .
 c02e29e8 uint64_t size = 0t3099832832
 c02e29f0 uint64_t p = 0t16540761088
 c02e29f8 uint64_t c = 0t1070318720
 c02e2a00 uint64_t c_min = 0t1070318720
 c02e2a08 uint64_t c_max = 0t1070318720
 . . .
 
 size is at 3GB, with c_max at 1GB.
 
 What gives? I'm looking at the code now, but was under the impression
 c_max would limit ARC growth. Granted, it's not a factor of 10, and
 it's certainly much better than the out-of-the-box growth to 24GB
 (this is a 32GB x4500), so clearly ARC growth is being limited, but it
 still grew to 3X c_max.
 
 Thanks,
 /jim
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
  
   
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

 
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
 
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] C'mon ARC, stay small...

2007-03-15 Thread Jim Mauro


Following a reboot:
 arc::print -tad
{
. . .
   c02e29e8 uint64_t size = 0t299008
   c02e29f0 uint64_t p = 0t16588228608
   c02e29f8 uint64_t c = 0t33176457216
   c02e2a00 uint64_t c_min = 0t1070318720
   c02e2a08 uint64_t c_max = 0t33176457216
. . .
}  
 c02e2a08 /Z 0x2000 --- set 
c_max to 512MB

arc+0x48:   0x7b9789000 =   0x2000
 arc::print -tad
{
. . .
   c02e29e8 uint64_t size = 0t299008
   c02e29f0 uint64_t p = 0t16588228608
   c02e29f8 uint64_t c = 0t33176457216
   c02e2a00 uint64_t c_min = 0t1070318720
   c02e2a08 uint64_t c_max = 0t536870912  - c_max is 512MB
. . .
}  
 ARC_mru::print -d size lsize

size = 0t294912
lsize = 0t32768


Run the workload a couple times...

   c02e29e8 uint64_t size = 0t27121205248 --- ARC size is 27GB
   c02e29f0 uint64_t p = 0t10551351442
   c02e29f8 uint64_t c = 0t27121332576
   c02e2a00 uint64_t c_min = 0t1070318720
   c02e2a08 uint64_t c_max = 0t536870912 - c_max is 512MB

 ARC_mru::print -d size lsize
size = 0t223985664
lsize = 0t221839360
 ARC_mfu::print -d size lsize
size = 0t26897219584  -- MFU list is almost 27GB ...
lsize = 0t26869121024

Thanks,
/jim




___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] C'mon ARC, stay small...

2007-03-15 Thread Jim Mauro

Will try that now...

/jim


[EMAIL PROTECTED] wrote:

I suppose I should have been more forward about making my last point.
If the arc_c_max isn't set in /etc/system, I don't believe that the ARC
will initialize arc.p to the correct value.   I could be wrong about
this; however, next time you set c_max, set c to the same value as c_max
and set p to half of c.  Let me know if this addresses the problem or
not.

-j

  
How/when did you configure arc_c_max?  
  

Immediately following a reboot, I set arc.c_max using mdb,
then verified reading the arc structure again.


arc.p is supposed to be
initialized to half of arc.c.  Also, I assume that there's a reliable
test case for reproducing this problem?
 
  

Yep. I'm using a x4500 in-house to sort out performance of a customer test
case that uses mmap. We acquired the new DIMMs to bring the
x4500 to 32GB, since the workload has a 64GB working set size,
and we were clobbering a 16GB thumper. We wanted to see how doubling
memory may help.

I'm trying clamp the ARC size because for mmap-intensive workloads,
it seems to hurt more than help (although, based on experiments up to this
point, it's not hurting a lot).

I'll do another reboot, and run it all down for you serially...

/jim



Thanks,

-j

On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote:
 
  
   


ARC_mru::print -d size lsize
 
  

size = 0t10224433152
lsize = 0t10218960896
   


ARC_mfu::print -d size lsize
 
  

size = 0t303450112
lsize = 0t289998848
   


ARC_anon::print -d size
 
  

size = 0
   
So it looks like the MRU is running at 10GB...


What does this tell us?

Thanks,
/jim



[EMAIL PROTECTED] wrote:
   


This seems a bit strange.  What's the workload, and also, what's the
output for:


 
  

ARC_mru::print size lsize
ARC_mfu::print size lsize
  
   


and

 
  

ARC_anon::print size
  
   


For obvious reasons, the ARC can't evict buffers that are in use.
Buffers that are available to be evicted should be on the mru or mfu
list, so this output should be instructive.

-j

On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote:

 
  

FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max:


  
   


arc::print -tad

 
  

{
. . .
 c02e29e8 uint64_t size = 0t10527883264
 c02e29f0 uint64_t p = 0t16381819904
 c02e29f8 uint64_t c = 0t1070318720
 c02e2a00 uint64_t c_min = 0t1070318720
 c02e2a08 uint64_t c_max = 0t1070318720
. . .

Perhaps c_max does not do what I think it does?

Thanks,
/jim


Jim Mauro wrote:
  
   


Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06
(update 3). All file IO is mmap(file), read memory segment, unmap, 
close.


Tweaked the arc size down via mdb to 1GB. I used that value because
c_min was also 1GB, and I was not sure if c_max could be larger than
c_minAnyway, I set c_max to 1GB.

After a workload run:

 
  

arc::print -tad
  
   


{
. . .
c02e29e8 uint64_t size = 0t3099832832
c02e29f0 uint64_t p = 0t16540761088
c02e29f8 uint64_t c = 0t1070318720
c02e2a00 uint64_t c_min = 0t1070318720
c02e2a08 uint64_t c_max = 0t1070318720
. . .

size is at 3GB, with c_max at 1GB.

What gives? I'm looking at the code now, but was under the impression
c_max would limit ARC growth. Granted, it's not a factor of 10, and
it's certainly much better than the out-of-the-box growth to 24GB
(this is a 32GB x4500), so clearly ARC growth is being limited, but it
still grew to 3X c_max.

Thanks,
/jim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

 
  

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
  
   


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
   


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
  

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] C'mon ARC, stay small...

2007-03-15 Thread Jim Mauro

All righty...I set c_max to 512MB, c to 512MB, and p to 256MB...

 arc::print -tad
{
...
   c02e29e8 uint64_t size = 0t299008
   c02e29f0 uint64_t p = 0t16588228608
   c02e29f8 uint64_t c = 0t33176457216
   c02e2a00 uint64_t c_min = 0t1070318720
   c02e2a08 uint64_t c_max = 0t33176457216
...
}
 c02e2a08 /Z 0x2000
arc+0x48:   0x7b9789000 =   0x2000
 c02e29f8 /Z 0x2000
arc+0x38:   0x7b9789000 =   0x2000
 c02e29f0 /Z 0x1000
arc+0x30:   0x3dcbc4800 =   0x1000
 arc::print -tad
{
...
   c02e29e8 uint64_t size = 0t299008
   c02e29f0 uint64_t p = 0t268435456  -- p is 
256MB
   c02e29f8 uint64_t c = 0t536870912  -- c is 
512MB

   c02e2a00 uint64_t c_min = 0t1070318720
   c02e2a08 uint64_t c_max = 0t536870912--- c_max is 512MB
...
}

After a few runs of the workload ...

 arc::print -d size
size = 0t536788992



Ah - looks like we're out of the woods. The ARC remains clamped at 512MB.

Thanks!
/jim


[EMAIL PROTECTED] wrote:

I suppose I should have been more forward about making my last point.
If the arc_c_max isn't set in /etc/system, I don't believe that the ARC
will initialize arc.p to the correct value.   I could be wrong about
this; however, next time you set c_max, set c to the same value as c_max
and set p to half of c.  Let me know if this addresses the problem or
not.

-j

  
How/when did you configure arc_c_max?  
  

Immediately following a reboot, I set arc.c_max using mdb,
then verified reading the arc structure again.


arc.p is supposed to be
initialized to half of arc.c.  Also, I assume that there's a reliable
test case for reproducing this problem?
 
  

Yep. I'm using a x4500 in-house to sort out performance of a customer test
case that uses mmap. We acquired the new DIMMs to bring the
x4500 to 32GB, since the workload has a 64GB working set size,
and we were clobbering a 16GB thumper. We wanted to see how doubling
memory may help.

I'm trying clamp the ARC size because for mmap-intensive workloads,
it seems to hurt more than help (although, based on experiments up to this
point, it's not hurting a lot).

I'll do another reboot, and run it all down for you serially...

/jim



Thanks,

-j

On Thu, Mar 15, 2007 at 06:57:12PM -0400, Jim Mauro wrote:
 
  
   


ARC_mru::print -d size lsize
 
  

size = 0t10224433152
lsize = 0t10218960896
   


ARC_mfu::print -d size lsize
 
  

size = 0t303450112
lsize = 0t289998848
   


ARC_anon::print -d size
 
  

size = 0
   
So it looks like the MRU is running at 10GB...


What does this tell us?

Thanks,
/jim



[EMAIL PROTECTED] wrote:
   


This seems a bit strange.  What's the workload, and also, what's the
output for:


 
  

ARC_mru::print size lsize
ARC_mfu::print size lsize
  
   


and

 
  

ARC_anon::print size
  
   


For obvious reasons, the ARC can't evict buffers that are in use.
Buffers that are available to be evicted should be on the mru or mfu
list, so this output should be instructive.

-j

On Thu, Mar 15, 2007 at 02:08:37PM -0400, Jim Mauro wrote:

 
  

FYI - After a few more runs, ARC size hit 10GB, which is now 10X c_max:


  
   


arc::print -tad

 
  

{
. . .
 c02e29e8 uint64_t size = 0t10527883264
 c02e29f0 uint64_t p = 0t16381819904
 c02e29f8 uint64_t c = 0t1070318720
 c02e2a00 uint64_t c_min = 0t1070318720
 c02e2a08 uint64_t c_max = 0t1070318720
. . .

Perhaps c_max does not do what I think it does?

Thanks,
/jim


Jim Mauro wrote:
  
   


Running an mmap-intensive workload on ZFS on a X4500, Solaris 10 11/06
(update 3). All file IO is mmap(file), read memory segment, unmap, 
close.


Tweaked the arc size down via mdb to 1GB. I used that value because
c_min was also 1GB, and I was not sure if c_max could be larger than
c_minAnyway, I set c_max to 1GB.

After a workload run:

 
  

arc::print -tad
  
   


{
. . .
c02e29e8 uint64_t size = 0t3099832832
c02e29f0 uint64_t p = 0t16540761088
c02e29f8 uint64_t c = 0t1070318720
c02e2a00 uint64_t c_min = 0t1070318720
c02e2a08 uint64_t c_max = 0t1070318720
. . .

size is at 3GB, with c_max at 1GB.

What gives? I'm looking at the code now, but was under the impression
c_max would limit ARC growth. Granted, it's not a factor of 10, and
it's certainly much better than the out-of-the-box growth to 24GB
(this is a 32GB x4500), so clearly ARC growth is being limited, but it
still grew to 3X c_max.

Thanks,
/jim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org