Re: ZFS tuning [was: hardware for home use large storage]

2010-02-17 Thread jhell 

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



On Tue, 16 Feb 2010 16:56, freebsd@ wrote:

On Wed, Feb 17, 2010 at 07:05:11AM +1100, Peter Jeremy wrote:

Overall, Barry's script makes an excellent proof-of-concept - which is
what he was offering.


You know, I had a verbose in-line response typed up, agreeing with some
points of yours and disagreeing with others (with perlfaq and other
reference material), but decided... fuck it!  :-)




Lol, (anyway)...

To all those involved in this.

More up-to-date: arc_summary.pl r182
I have done some more minor formatting with the heading section.

I have added some L2 ARC stats.

Adjusted called binaries to have full paths for the moment until I get 
around to fixing it a little more. Not having the full path caused some 
frustrations with emailing the output while running from cron. If this 
will be a problem with anyone running it and you come across errors where 
something can not be executed please be patient as I will have this 
adjusted soon.


arcstat.pl: Please update the WiKi with the missing URL, I uploaded the 
original FreeBSD modified copy of arcstat.pl script to the same location 
as arc_summary.pl in case anyone wants to grab that file for FreeBSD while 
they are their. I have also signed this file with my public key. I won't 
be making any mods to this script as I see it works as intended. I may be 
persuaded in the future to make adjustments to keep it working and small 
bug fixes.


http://jhell.googlecode.com/files/arc_summary.pl
http://jhell.googlecode.com/files/arcstat.pl

MD5 (arc_summary.pl) = 6587be39266bd131ed5f4611f321c9e2
SHA256 (arc_summary.pl) = 
fea99963562e1caee01d6ccd9af1b592cb18fcfaf81308bd1ee21c546c8290ad
SIZE (arc_summary.pl) = 12875

MD5 (arcstat.pl) = 9b7d7b4d52ba997c7c59bc0afaca8aa9
SHA256 (arcstat.pl) = 
b97fc59460468bb2bab6d53009e67f2de87fd5817e7b5656266ea4fd75079b2a
SIZE (arcstat.pl) = 8172

Best regards.

And watching for replies,

- -- 


 jhell

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.14 (FreeBSD)

iQEcBAEBAgAGBQJLe6G9AAoJEJBXh4mJ2FR++mYIAJXgJ468WX65PErzYiNySmB/
Sliz5Qur7kpqeP/LHuJHXk/FG+JaaQJQbh/9vIsX2zoqnBK7gqyqTOdv2pw+sKff
PNcwdhYbJXEYTnGorj4FXmEI4IkXfuYM281nEpytSVU1CLTnL8oeWU9YRFTw0Fhh
Gtvcy2jSnOmjjARt/0Ot1PKfOE9/PTghBTjvmGJIUz11evKsacZmXEIpv1P3KnS9
XMmeDjARkMpXCqDsBPPWeD/YYWUW6SA3Iq+BP0CeVqk7ytQ/q2WI7wH5pC+EscTx
dLczky3GR5TeeHrRVQCKWVzQGoxQFH85X2MGipgOae6viHHC+Nk2J1cKVWhLx5s=
=zRvh
-END PGP SIGNATURE-
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-17 Thread Bartosz Stec 

On 2010-02-17 08:58, jhell wrote:


To all those involved in this.

More up-to-date: arc_summary.pl r182


Best regards.

And watching for replies,


So here's my reply (last line seems most interesting ;) :

# ./arc_summary.pl
-
System Summary śro 17 lut 2010 08:16:37 UTC

FreeBSD 9.0-CURRENT #20: Fri Feb 12 19:21:34 CET 2010 ncpnc

Kernel Version: 99 (osreldate)

Hardware Platform: i386
Processor Architecture: i386

Storage pool Version: 14
Filesystem Version: 3

Physical RAM: 1523.41 MiB


Kernel Memory Usage
TEXT: 9782828, 9.33 MiB
DATA: 216801280, 206.76 MiB
TOTAL: 226584108, 216.09 MiB

9:16 up 4 days, 6:54, 1 user, load averages: 0,47 0,92 0,59
-

ARC Summary

Meta Limit: 120.00 MiB
Meta Used: 135.00% 162.00 MiB

Throttle Count: 0

ARC Size:
Current Size: 190.03 MiB (arcsize)
Target Size (Adaptive): 209.12 MiB (c)
Min Size (Hard Limit): 60.00 MiB (arc_min)
Max Size (Hard Limit): 480.00 MiB (arc_max)

ARC Size Breakdown:
Recently Used Cache Size: 16.00% 33.46 MiB (p)
Frequently Used Cache Size: 84.00% 175.66 MiB (c-p)

ARC Efficiency:
Cache Access Total: 28459411
Cache Hit Ratio: 94.57% 26913166
Cache Miss Ratio: 5.43% 1546245
Actual Hit Ratio: 81.81% 23282481

Data Demand Efficiency: 99.48%
Data Prefetch Efficiency: 54.15%

CACHE HITS BY CACHE LIST:
Anonymous: 11.07% 2978474
Most Recently Used: 16.42% 4419712 (mru)
Most Frequently Used: 70.09% 18862769 (mfu)
Most Recently Used Ghost: 0.98% 264205 (mru_ghost)
Most Frequently Used Ghost: 1.44% 388006 (mfu_ghost)

CACHE HITS BY DATA TYPE:
Demand Data: 53.58% 14419664
Prefetch Data: 0.07% 20017
Demand Metadata: 32.68% 8794182
Prefetch Metadata: 13.67% 3679303

CACHE MISSES BY DATA TYPE:
Demand Data: 4.88% 75410
Prefetch Data: 1.10% 16946
Demand Metadata: 59.18% 915133
Prefetch Metadata: 34.84% 538756


L2 ARC Summary

Low Memory Aborts: 0
Bad Checksums: 0
IO Errors: 0

L2 ARC Size:
Current Size: 0.00 MiB
Header Size: 0.00 MiB

L2 ARC Breakdown:
L2 Access Total: 0
Illegal division by zero at ./arc_summary.pl line 242.

--
 Bartosz Stec


___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-17 Thread Torfinn Ingolfsen 
On Wed, 17 Feb 2010 09:19:49 +0100
Bartosz Stec bartosz.s...@it4pro.pl wrote:

 So here's my reply (last line seems most interesting ;) :
[...snipped...]
 Illegal division by zero at ./arc_summary.pl line 242.

FWIW, I also got this line when I ran this script on my idle zfs server.
-- 
Torfinn

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-17 Thread jhell 


On Tue, 16 Feb 2010 12:06, bp@ wrote:

On 2/15/10 7:49 PM, jhell wrote:


As I make final modifications to the script I will keep the below URLs
updated and welcome any bug reports or modification requests to me
personally.

Here is the URLs:
http://jhell.googlecode.com/files/arc_summary.pl
http://jhell.googlecode.com/files/arc_summary.pl.asc


Nice.  How about including relevant lines from /boot/loader.conf, maybe 
something like this tacked on the end of the script (excuse my Perl, I'm a 
Python guy).



 Loader Settings #
open(LOADER, '/boot/loader.conf');
print \n/boot/loader.conf settings:\n;
while (LOADER){
   chomp;
   if (/^\s*(zfs|vfs\.zfs|vm\.kmem)/){
   print \t$_\n;
   }
}


Yes, it should more or less duplicate the sysctl values, but it may make it 
more obvious where the settings are coming from, or if the user has bad or 
ignored settings


Barry



Hi Barry,

	This is a very nice idea so please do not get me wrong when I say 
that I would rather stay away from having to open config files and start 
comparing results. I don't feel that this is in the place of the script to 
do this type of action and can lead to confusion upon the users behalf.


sysctl(8) is by far better off getting the values from a running system 
rather than parsing values obtained from a file that does not change the 
system except for after a reboot upon which the values outlined in 
loader.conf can be obtained from sysctl mibs.


The latest version of the script that I have uploaded has the sysctl mibs 
spilled out at the bottom. These are mibs that I felt were relevent to the 
tuning of ZFS and FreeBSD obtained from various sources and memory recall 
so I might have missed a few so if you or anyone else has a mib to be 
added please forward them to me in a email and I will be sure to add it.


Regards, and thanks for the feedback.

--

 jhell

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-17 Thread jhell 


On Tue, 16 Feb 2010 12:59, freebsd@ wrote:

On Tue, Feb 16, 2010 at 11:06:43AM -0600, Barry Pederson wrote:

On 2/15/10 7:49 PM, jhell wrote:


As I make final modifications to the script I will keep the below URLs
updated and welcome any bug reports or modification requests to me
personally.

Here is the URLs:
http://jhell.googlecode.com/files/arc_summary.pl
http://jhell.googlecode.com/files/arc_summary.pl.asc


Nice.  How about including relevant lines from /boot/loader.conf,
maybe something like this tacked on the end of the script (excuse my
Perl, I'm a Python guy).


 Loader Settings #
open(LOADER, '/boot/loader.conf');
print \n/boot/loader.conf settings:\n;
while (LOADER){
chomp;
if (/^\s*(zfs|vfs\.zfs|vm\.kmem)/){
print \t$_\n;
}
}


Yes, it should more or less duplicate the sysctl values, but it may
make it more obvious where the settings are coming from, or if the
user has bad or ignored settings


Major problems with the above code:

1) Opens /boot/loader.conf for rw access; should be read-only
2) Makes the assumption /boot/loader.conf exists
3) Does not close the fd
4) Excessively quotes variables for no justified reason
5) Makes some bad assumptions about the contents of the file (ex.
  comments with the word zfs in them would match)

The code should really be something like what's below.  This should
be much more manageable as well (@tunables that is), although I always
worry when using grep()...




Very nice!,

Ill keep this for reference later on. This might just come in handy at 
some point.


But for the sake of arc_summary.pl I feel this is beyond the scope of what 
its intended use is. See previous email in response to Barry.


Thanks Jeremy

--

 jhell

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-17 Thread Bartosz Stec 

On 2010-02-17 09:32, Torfinn Ingolfsen wrote:

On Wed, 17 Feb 2010 09:19:49 +0100
Bartosz Stecbartosz.s...@it4pro.pl  wrote:

   

So here's my reply (last line seems most interesting ;) :
 

[...snipped...]
   

Illegal division by zero at ./arc_summary.pl line 242.
 

FWIW, I also got this line when I ran this script on my idle zfs server.
   
I'm not a PERL programmer (or programmer at all ;), but what I see is 
script doesn't check if L2ARC is used at all, so it will always try 
compute these lines:


printf(\tL2 Hit Ratio:\t\t\t%0.2f%%\t%d\n, 100 * ( $l2_hits / ( 
$l2_hits + $l2_misses )), $l2_hits );
printf(\tL2 Miss Ratio:\t\t\t%0.2f%%\t%d\n, 100 * ( $l2_misses / ( 
$l2_hits + $l2_misses )), $l2_misses );
printf(\tL2 Feeds Ratio:\t\t\t%0.2f%%\t%d\n, 100 * ( $l2_feeds / ( 
$l2_hits + $l2_misses )), $l2_feeds );


Without active L2ARC it will always generate divide at zeo error, so it 
seems that additional check for usable L2ARC values is needed at first 
place.


--
Bartosz Stec


___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-17 Thread Scot Hetzel 
On Wed, Feb 17, 2010 at 2:56 AM, Bartosz Stec bartosz.s...@it4pro.pl wrote:
 On 2010-02-17 09:32, Torfinn Ingolfsen wrote:

 On Wed, 17 Feb 2010 09:19:49 +0100
 Bartosz Stecbartosz.s...@it4pro.pl  wrote:



 So here's my reply (last line seems most interesting ;) :


 [...snipped...]


 Illegal division by zero at ./arc_summary.pl line 242.


 FWIW, I also got this line when I ran this script on my idle zfs server.


 I'm not a PERL programmer (or programmer at all ;), but what I see is script
 doesn't check if L2ARC is used at all, so it will always try compute these
 lines:

 printf(\tL2 Hit Ratio:\t\t\t%0.2f%%\t%d\n, 100 * ( $l2_hits / ( $l2_hits +
 $l2_misses )), $l2_hits );
 printf(\tL2 Miss Ratio:\t\t\t%0.2f%%\t%d\n, 100 * ( $l2_misses / (
 $l2_hits + $l2_misses )), $l2_misses );
 printf(\tL2 Feeds Ratio:\t\t\t%0.2f%%\t%d\n, 100 * ( $l2_feeds / (
 $l2_hits + $l2_misses )), $l2_feeds );

 Without active L2ARC it will always generate divide at zeo error, so it
 seems that additional check for usable L2ARC values is needed at first
 place.


The attached patch fixes the divide by zero errors.

Scot


patch1
Description: Binary data
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-17 Thread jhell 

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



On Wed, 17 Feb 2010 04:53, jhell@ wrote:

 PGP Command Output 
gpg: Signature made Wed Feb 17 04:53:27 2010 EST using RSA key ID 89D8547E
gpg: Good signature from jhell jh...@dataix.net
--- Begin PGP Signed Message Verified 2010-02-17 06:13:53 --



On Wed, 17 Feb 2010 03:56, bartosz.stec@ wrote:

On 2010-02-17 09:32, Torfinn Ingolfsen wrote:

On Wed, 17 Feb 2010 09:19:49 +0100
Bartosz Stecbartosz.s...@it4pro.pl  wrote:



So here's my reply (last line seems most interesting ;) :


[...snipped...]


Illegal division by zero at ./arc_summary.pl line 242.


FWIW, I also got this line when I ran this script on my idle zfs server.

I'm not a PERL programmer (or programmer at all ;), but what I see is 
script

doesn't check if L2ARC is used at all, so it will always try compute these
lines:

printf(\tL2 Hit Ratio:\t\t\t%0.2f%%\t%d\n, 100 * ( $l2_hits / ( $l2_hits 
+

$l2_misses )), $l2_hits );
printf(\tL2 Miss Ratio:\t\t\t%0.2f%%\t%d\n, 100 * ( $l2_misses / ( 
$l2_hits

+ $l2_misses )), $l2_misses );
printf(\tL2 Feeds Ratio:\t\t\t%0.2f%%\t%d\n, 100 * ( $l2_feeds / ( 
$l2_hits

+ $l2_misses )), $l2_feeds );

Without active L2ARC it will always generate divide at zeo error, so it 
seems

that additional check for usable L2ARC values is needed at first place.




Thanks for reporting this. As I am usually on a system that is using a
L2ARC I wouldn't have noticed it.

I should have this fixed in about 10 hours. But as I am writing this email
I am heading off to bed, work calls in the morning.

Check back tomorrow night for a updated version and adjust the current to
your liking for the moment.

;) Thanks again.

--

jhell


 End PGP Signed Message Verified 2010-02-17 06:13:53 ---



I take that back.

I just uploaded a modified version that I wrapped with a if statement to 
check and see if l2_hits = 0.


If you don't have a L2ARC you will now get a message explaining why its not 
included in your summary.


I couldn't sleep until I fixed this knowing that more people are probably 
going to come across this problem and email back.


New rev: 184

MD5 (arc_summary.pl) = f47bac165e7bf707d5f81cfdd007c30a
SHA256 (arc_summary.pl) = 
794dce069ff649598d99204b362d141a19da47dcf60ec165b260d55a5c9d493f
SIZE (arc_summary.pl) = 12695

http://jhell.googlecode.com/files/arc_summary.pl
http://jhell.googlecode.com/files/arc_summary.pl.asc

Now I can finally sleep ;)

- -- 


 jhell

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.14 (FreeBSD)

iQEcBAEBAgAGBQJLe9BrAAoJEJBXh4mJ2FR+0K8IAKA43hk95Kll9mLfMWj5bUPp
ZLlDzZPPy30Ign6wfbSO0wImLW0UVa9wAL0EwWb78F9T/3AJ2fQZFgWrOp/t+eV4
iKG8rsEy6t6YDYYZ7G6XnSibiCO+M+L+b6eSWMbcl/Ak8n+1PZUQisFevq/K0cCu
31ktjNxC6eqK1s0rKn/CgyXKO/rga60U12OHG9SLInM8J1dtHSGAp6kBO0B6C9+m
uzEKOkUxXlYZpo+vlR9alByPWfiG9JqkgiYcOeXcgo0kb405cVT5jwBrOY9UnTb8
phxY6RXUViGP/quX2P+tGIYO47gDvBiGY/XRyTO1bmM+O0nPTnnKHpJg9NBvZ/g=
=OFDn
-END PGP SIGNATURE-
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-17 Thread jhell 

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



Due to the nature of this thread and its current list involvement I am 
going to be starting a new thread in freebsd-filesystems@ just for 
arc_summary.pl tomorrow night with the subject below.


[arc_summary.pl] Adjustments, PR  Requests

I would appreciate if you would join me there for any response or problems 
that may arise for the use of this script. I don't feel that this list is 
any longer the correct place to hold this discussion as it is more ZFS 
centric than stable@ was meant for.


If someone would like to start the subject off before I get to it please 
feel free.


Thanks in advance.

- -- 


 jhell

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.14 (FreeBSD)

iQEcBAEBAgAGBQJLe9OTAAoJEJBXh4mJ2FR+LdwH/0cCwnsWogtHfGyU4PJcu1fG
PV/PxpgMKQsTIfckGI8ef/UPCG7lki6FD9+bPrsOrFy2ZuvGhK+jfxXsa+99Rt4h
ovi6qopLjfHde9Ztd87E3Ds88kR0KEszdnDvbg7mr/xAz9V7hFrLWvmIqxaedg06
I+j932tUfFmA2xiExBWVjtHfdYD2YaO671SET0nPSGz0b6yRlfedOvTZvheI+l05
fB1pXQNwve1en6oDtBBb0pgoODI9HgrZyrQbJzTFH/hXremdO5Dv5AUULI/XKwNA
KSSWRMRKYODdX7cDIDbS2+7GjLj1V85gdBc10usR4AqpSFl8aGmTIUfK187y+qs=
=7QZB
-END PGP SIGNATURE-
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-16 Thread Alexander Leidinger 

Quoting jhell jh...@dataix.net (from Mon, 15 Feb 2010 20:49:38 -0500):


On Mon, 15 Feb 2010 16:20, 000.fbsd@ wrote:



I think you are referring to this script:
http://cuddletech.com/arc_summary/
and its FreeBSD version
http://bitbucket.org/koie/arc_summary/changeset/dbe14d2cf52b/



Another useful script is arcstat.pl from
http://blogs.sun.com/realneel/entry/zfs_arc_statistics
There are FreeBSD version floating around but I can't find a link  
to it, so I am sending it as attachment.


It would be nice to have some standardized scripts for monitoring  
 debugging ZFS issues attached to FreeBSD Wiki page about ZFS  
tuning. Then everebody will use the same scripts, same output  
format. It will be easier to compare results in discussions etc.


So if anybody of you have write permissions to Wiki, can you add  
those two scripts? (or make some better ;])


It is funny that you guys are all of a sudden talking about this, as  
I was just working on some modifications to the arc_summary.pl  
script for some better formatting and inclusion of kmem statistics.



Here is the URLs:
http://jhell.googlecode.com/files/arc_summary.pl
http://jhell.googlecode.com/files/arc_summary.pl.asc


I added references to all those scripts to the wiki. I also improved  
(I hope) the some parts. Please have a look and feel free to submit  
improvements (or register with FirstnameLastname and tell me about it,  
this way I'm not the bottleneck for changes).


Bye,
Alexander.

--
I'm not stupid, I'm not expendable, and I'M NOT GOING!

http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org   netchild @ FreeBSD.org  : PGP ID = 72077137
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-16 Thread Barry Pederson 

On 2/15/10 7:49 PM, jhell wrote:


As I make final modifications to the script I will keep the below URLs
updated and welcome any bug reports or modification requests to me
personally.

Here is the URLs:
http://jhell.googlecode.com/files/arc_summary.pl
http://jhell.googlecode.com/files/arc_summary.pl.asc


Nice.  How about including relevant lines from /boot/loader.conf, maybe 
something like this tacked on the end of the script (excuse my Perl, I'm 
a Python guy).



 Loader Settings #
open(LOADER, '/boot/loader.conf');
print \n/boot/loader.conf settings:\n;
while (LOADER){
chomp;
if (/^\s*(zfs|vfs\.zfs|vm\.kmem)/){
print \t$_\n;
}
}


Yes, it should more or less duplicate the sysctl values, but it may make 
it more obvious where the settings are coming from, or if the user has 
bad or ignored settings


Barry
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-16 Thread Jeremy Chadwick 
On Tue, Feb 16, 2010 at 11:06:43AM -0600, Barry Pederson wrote:
 On 2/15/10 7:49 PM, jhell wrote:
 
 As I make final modifications to the script I will keep the below URLs
 updated and welcome any bug reports or modification requests to me
 personally.
 
 Here is the URLs:
 http://jhell.googlecode.com/files/arc_summary.pl
 http://jhell.googlecode.com/files/arc_summary.pl.asc
 
 Nice.  How about including relevant lines from /boot/loader.conf,
 maybe something like this tacked on the end of the script (excuse my
 Perl, I'm a Python guy).
 
 
  Loader Settings #
 open(LOADER, '/boot/loader.conf');
 print \n/boot/loader.conf settings:\n;
 while (LOADER){
 chomp;
 if (/^\s*(zfs|vfs\.zfs|vm\.kmem)/){
 print \t$_\n;
 }
 }
 
 
 Yes, it should more or less duplicate the sysctl values, but it may
 make it more obvious where the settings are coming from, or if the
 user has bad or ignored settings

Major problems with the above code:

1) Opens /boot/loader.conf for rw access; should be read-only
2) Makes the assumption /boot/loader.conf exists
3) Does not close the fd
4) Excessively quotes variables for no justified reason
5) Makes some bad assumptions about the contents of the file (ex.
   comments with the word zfs in them would match)

The code should really be something like what's below.  This should
be much more manageable as well (@tunables that is), although I always
worry when using grep()...

-- 
| Jeremy Chadwick   j...@parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

#!/usr/local/bin/perl

my @tunables = qw(
vfs\.zfs\..+
vm\.kmem_size.*
);

my $loaderconf = /boot/loader.conf;

if (-e $loaderconf)
{
  open(FH, , $loaderconf) or die;
  while (FH)
  {
# Get rid of trailing newlines and preceding spaces.
chomp;
s/^\s+//g;

# Match against any key=value pair, and then see if the
# key portion matches against any regex string in @tunables.
if (/^([\w\.]+)=.+/)
{
if (grep $1 =~ /$_/, @tunables) {
print \t, $_ , \n;
}
}
  }
  close(FH);
}

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-16 Thread Matt Reimer 
On Mon, Feb 15, 2010 at 5:49 PM, jhell jh...@dataix.net wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 It is funny that you guys are all of a sudden talking about this, as I was
 just working on some modifications to the arc_summary.pl script for some
 better formatting and inclusion of kmem statistics.

 My intent on the modifications is to make the output more usable to the
 whole community by revealing the relevant system information that can be
 included in an email to the lists for diagnosis by others.

...

 Example output:
 - -
 System Summary
 OS Revision:199506
 OS Release Date:703100
 Hardware Platform:  i386
 Processor Architecture: i386
 Storage pool Version:   13
 Filesystem Version: 3

 Kernel Memory Usage
 TEXT:   8950208 KiB,8 MiB
 DATA:   206347264 KiB,  196 MiB
 TOTAL:  215297472 KiB,  205 MiB


Above did you really mean 8950208 B not KiB, etc.?

Matt
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-16 Thread Jeremy Chadwick 
On Wed, Feb 17, 2010 at 07:05:11AM +1100, Peter Jeremy wrote:
 Overall, Barry's script makes an excellent proof-of-concept - which is
 what he was offering.

You know, I had a verbose in-line response typed up, agreeing with some
points of yours and disagreeing with others (with perlfaq and other
reference material), but decided... fuck it!  :-)

-- 
| Jeremy Chadwick   j...@parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-16 Thread jhell 


On Tue, 16 Feb 2010 13:30, mattjreimer@ wrote:

On Mon, Feb 15, 2010 at 5:49 PM, jhell jh...@dataix.net wrote:


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

It is funny that you guys are all of a sudden talking about this, as I was
just working on some modifications to the arc_summary.pl script for some
better formatting and inclusion of kmem statistics.

My intent on the modifications is to make the output more usable to the
whole community by revealing the relevant system information that can be
included in an email to the lists for diagnosis by others.


...


Example output:
- -
System Summary
OS Revision:199506
OS Release Date:703100
Hardware Platform:  i386
Processor Architecture: i386
Storage pool Version:   13
Filesystem Version: 3

Kernel Memory Usage
TEXT:   8950208 KiB,8 MiB
DATA:   206347264 KiB,  196 MiB
TOTAL:  215297472 KiB,  205 MiB



Above did you really mean 8950208 B not KiB, etc.?

Matt



Yes, Thank you for pointing this out.

I have fixed this and it will be added to the the same url as before in 
about 3 or so hours from the time of this email. This update will also add 
some stats for L2 ARC as well.


Thanks again.

--

 jhell

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-15 Thread Miroslav Lachman 

Alexander Leidinger wrote:

[...]


kstat.zfs.misc.arcstats.c
kstat.zfs.misc.arcstats.c_min
kstat.zfs.misc.arcstats.c_max


c_max is vfs.zfs.arc_max, c_min is vfs.zfs.arc_min.


kstat.zfs.misc.arcstats.evict_skip
kstat.zfs.misc.arcstats.memory_throttle_count
kstat.zfs.misc.arcstats.size


I'm not very sure about size and c... both represent some kind of
current size, but they are not the same.


About the tuning I would recommend to depend upon a more human readable
representation. I've seen someone posting something like this, but I do
not know how it was generated (some kind of script, but I do not know
where to get it).


I think you are referring to this script:
http://cuddletech.com/arc_summary/
and its FreeBSD version
http://bitbucket.org/koie/arc_summary/changeset/dbe14d2cf52b/

It gives output like this:

# arc_summary.sh

System Memory:
 Physical RAM:  4978 MB
 Free Memory :  755 MB

ARC Size:
 Current Size: 1028 MB (arcsize)
 Target Size (Adaptive):   1028 MB (c)
 Min Size (Hard Limit):50 MB (zfs_arc_min)
 Max Size (Hard Limit):1205 MB (zfs_arc_max)

ARC Size Breakdown:
 Most Recently Used Cache Size:  93%963 MB (p)
 Most Frequently Used Cache Size: 6%65 MB (c-p)

ARC Efficency:
 Cache Access Total: 358720716
 Cache Hit Ratio:  97%   350351031  [Defined State 
for buffer]
 Cache Miss Ratio:  2%   8369685[Undefined 
State for Buffer]

 REAL Hit Ratio:   76%   272917080  [MRU/MFU Hits Only]

 Data Demand   Efficiency:96%
 Data Prefetch Efficiency:27%

CACHE HITS BY CACHE LIST:
  Anon:   22%77179355 
 [ New Customer, First Cache Hit ]
  Most Recently Used: 45%158252587 (mru) 
 [ Return Customer ]
  Most Frequently Used:   32%114664493 (mfu) 
 [ Frequent Customer ]
  Most Recently Used Ghost:0%9777 (mru_ghost) 
 [ Return Customer Evicted, Now Back ]
  Most Frequently Used Ghost:  0%244819 (mfu_ghost) 
 [ Frequent Customer Evicted, Now Back ]

CACHE HITS BY DATA TYPE:
  Demand Data: 1%4375918
  Prefetch Data:   0%150148
  Demand Metadata:76%267830502
  Prefetch Metadata:  22%77994463
CACHE MISSES BY DATA TYPE:
  Demand Data: 1%135956
  Prefetch Data:   4%400434
  Demand Metadata:73%6177748
  Prefetch Metadata:  19%1655547
-


Another useful script is arcstat.pl from
http://blogs.sun.com/realneel/entry/zfs_arc_statistics
There are FreeBSD version floating around but I can't find a link to it, 
so I am sending it as attachment.


It would be nice to have some standardized scripts for monitoring  
debugging ZFS issues attached to FreeBSD Wiki page about ZFS tuning. 
Then everebody will use the same scripts, same output format. It will be 
easier to compare results in discussions etc.


So if anybody of you have write permissions to Wiki, can you add those 
two scripts? (or make some better ;])


Understanding to tuning of ZFS is really hard with lack of documentation ;(

Miroslav Lachman
#!/usr/bin/perl -w
#
# Print out ZFS ARC Statistics exported via kstat(1)
# For a definition of fields, or usage, use arctstat.pl -v
#
# Author: Neelakanth Nadgir http://blogs.sun.com/realneel
# Comments/Questions/Feedback to neel_sun.com or neel_gnu.org
#
# CDDL HEADER START
# 
# The contents of this file are subject to the terms of the
# Common Development and Distribution License, Version 1.0 only
# (the License).  You may not use this file except in compliance
# with the License.
# 
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
# 
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets [] replaced with your own identifying
# information: Portions Copyright [] [name of copyright owner]
# 
# CDDL HEADER END
#
#
# Fields have a fixed width. Every interval, we fill the v
# hash with its corresponding value (v[field]=value) using calculate(). 
# @hdr is the array of fields that needs to be printed, so we
# just iterate over this array and print the values using our pretty printer.

use strict;
use POSIX qw(strftime);
#use Sun::Solaris::Kstat;
use Getopt::Long;
use IO::Handle;

my %cols = (# HDR = [Size, Description]
Time  =[8, Time],
hits  =[4,

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-15 Thread jhell 

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



On Mon, 15 Feb 2010 16:20, 000.fbsd@ wrote:

Alexander Leidinger wrote:

[...]


kstat.zfs.misc.arcstats.c
kstat.zfs.misc.arcstats.c_min
kstat.zfs.misc.arcstats.c_max


c_max is vfs.zfs.arc_max, c_min is vfs.zfs.arc_min.


kstat.zfs.misc.arcstats.evict_skip
kstat.zfs.misc.arcstats.memory_throttle_count
kstat.zfs.misc.arcstats.size


I'm not very sure about size and c... both represent some kind of
current size, but they are not the same.


About the tuning I would recommend to depend upon a more human readable
representation. I've seen someone posting something like this, but I do
not know how it was generated (some kind of script, but I do not know
where to get it).


I think you are referring to this script:
http://cuddletech.com/arc_summary/
and its FreeBSD version
http://bitbucket.org/koie/arc_summary/changeset/dbe14d2cf52b/

It gives output like this:

# arc_summary.sh

System Memory:
Physical RAM:  4978 MB
Free Memory :  755 MB

ARC Size:
Current Size: 1028 MB (arcsize)
Target Size (Adaptive):   1028 MB (c)
Min Size (Hard Limit):50 MB (zfs_arc_min)
Max Size (Hard Limit):1205 MB (zfs_arc_max)

ARC Size Breakdown:
Most Recently Used Cache Size:  93%963 MB (p)
Most Frequently Used Cache Size: 6%65 MB (c-p)

ARC Efficency:
Cache Access Total: 358720716
Cache Hit Ratio:  97%   350351031  [Defined State for 
buffer]
Cache Miss Ratio:  2%   8369685[Undefined State for 
Buffer]

REAL Hit Ratio:   76%   272917080  [MRU/MFU Hits Only]

Data Demand   Efficiency:96%
Data Prefetch Efficiency:27%

   CACHE HITS BY CACHE LIST:
 Anon:   22%77179355  [ New Customer, 
First Cache Hit ]
 Most Recently Used: 45%158252587 (mru)  [ Return 
Customer ]
 Most Frequently Used:   32%114664493 (mfu)  [ Frequent 
Customer ]
 Most Recently Used Ghost:0%9777 (mru_ghost)  [ Return 
Customer Evicted, Now Back ]
 Most Frequently Used Ghost:  0%244819 (mfu_ghost)  [ 
Frequent Customer Evicted, Now Back ]

   CACHE HITS BY DATA TYPE:
 Demand Data: 1%4375918
 Prefetch Data:   0%150148
 Demand Metadata:76%267830502
 Prefetch Metadata:  22%77994463
   CACHE MISSES BY DATA TYPE:
 Demand Data: 1%135956
 Prefetch Data:   4%400434
 Demand Metadata:73%6177748
 Prefetch Metadata:  19%1655547
-


Another useful script is arcstat.pl from
http://blogs.sun.com/realneel/entry/zfs_arc_statistics
There are FreeBSD version floating around but I can't find a link to it, so I 
am sending it as attachment.


It would be nice to have some standardized scripts for monitoring  
debugging ZFS issues attached to FreeBSD Wiki page about ZFS tuning. Then 
everebody will use the same scripts, same output format. It will be easier to 
compare results in discussions etc.


So if anybody of you have write permissions to Wiki, can you add those two 
scripts? (or make some better ;])


Understanding to tuning of ZFS is really hard with lack of documentation ;(

Miroslav Lachman



It is funny that you guys are all of a sudden talking about this, as I was 
just working on some modifications to the arc_summary.pl script for some 
better formatting and inclusion of kmem statistics.


My intent on the modifications is to make the output more usable to the 
whole community by revealing the relevant system information that can be 
included in an email to the lists for diagnosis by others.


Previously the output of the script was a little bit groggy, had long 
lines and did not include relevant other system information.


Currently I am working on cleaning up the code a little and moving the ZFS 
Tunable section to the bottom of the output where it will actually 
contain the values of the sysctl's instead of just being a blank list.


I would certainly appreciate any feedback I could get from this  other 
things you think might be relevant in its output.


Thanks for bringing this subject up.

As I make final modifications to the script I will keep the below URLs 
updated and welcome any bug reports or modification requests to me 
personally.


Here is the URLs:
http://jhell.googlecode.com/files/arc_summary.pl
http://jhell.googlecode.com/files/arc_summary.pl.asc

MD5 (arc_summary.pl) = bff13dcf119ff979d9aa52b3d8ae53b9
SHA256 (arc_summary.pl) = 
a29260946760a89614f888d53d0f188fb24bcd96acd5e0917604d494ed843ada
SIZE (arc_summary.pl) = 9453


Example output:
-

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-15 Thread Matthew D. Fuller 
On Mon, Feb 15, 2010 at 08:49:38PM -0500 I heard the voice of
jhell, and lo! it spake thus:
 
 As I make final modifications to the script I will keep the below
 URLs updated and welcome any bug reports or modification requests to
 me personally.

Well, here's one:

 OS Revision:  199506

There's no reason to show this; it's just confusing because it'll be
misinterpreted.  kern.osrevision isn't what you probably think it is.
It's just the BSD #define in param.h, which (aside from a blip which
was instantly reverted) last changed in 1996 when the -Lite2 import
was done.


-- 
Matthew Fuller (MF4839)   |  fulle...@over-yonder.net
Systems/Network Administrator |  http://www.over-yonder.net/~fullermd/
   On the Internet, nobody can hear you scream.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-15 Thread jhell 


On Mon, 15 Feb 2010 21:20, fullermd@ wrote:

On Mon, Feb 15, 2010 at 08:49:38PM -0500 I heard the voice of
jhell, and lo! it spake thus:


As I make final modifications to the script I will keep the below
URLs updated and welcome any bug reports or modification requests to
me personally.


Well, here's one:


OS Revision:199506


There's no reason to show this; it's just confusing because it'll be
misinterpreted.  kern.osrevision isn't what you probably think it is.
It's just the BSD #define in param.h, which (aside from a blip which
was instantly reverted) last changed in 1996 when the -Lite2 import
was done.



Thanks!, No I did not have any understanding of that till this moment but 
had included it just for completeness. In that case I will mark that for 
deletion.


Regards,

--

 jhell

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS tuning [was: hardware for home use large storage]

2010-02-15 Thread jhell 


On Mon, 15 Feb 2010 21:20, fullermd@ wrote:

On Mon, Feb 15, 2010 at 08:49:38PM -0500 I heard the voice of
jhell, and lo! it spake thus:


As I make final modifications to the script I will keep the below
URLs updated and welcome any bug reports or modification requests to
me personally.


Well, here's one:


OS Revision:199506


There's no reason to show this; it's just confusing because it'll be
misinterpreted.  kern.osrevision isn't what you probably think it is.
It's just the BSD #define in param.h, which (aside from a blip which
was instantly reverted) last changed in 1996 when the -Lite2 import
was done.



Removed in revision 171, and added output for sysctl tunables to the 
bottom.


Current branches or exact matches of sysctl's that are included are...
kern.maxusers
vfs.zfs
vm.kmem_size
vm.kmem_size_max

If there is more sysctl's that you think should be added please let me 
know and I will add and update the script.


The new revision(171) is in the same url as before.

MD5 (arc_summary.pl) = 29b276a6e2f13eedf5d36370994b7f0e
SHA256 (arc_summary.pl) = 
15a27b9eb71eddd64ee07a515c136f8467783dfb1075d9707028a082387c5127
SIZE (arc_summary.pl) = 9449

Regards,

--

 jhell

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org