Re: [zfs-discuss] How to find poor performing disks

[Simon Gao]

Running "iostat -nxce 1", I saw write sizes alternate between two raidz groups 
in the same pool.  

At one time, drives on cotroller 1 have larger writes (3-10 times) than ones on 
controller2:

extended device statistics    errors ---

r/sw/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b s/w h/w trn tot 
device 
0.00.00.00.0  0.0  0.00.00.0   0   0   0   0   0   0 
fd0
0.00.00.00.0  0.0  0.00.00.0   0   0   2   0   0   2 
c1t1d0 
0.00.00.00.0  0.0  0.00.00.0   0   0   1   0   0   1 
c0t10d0
0.00.00.00.0  0.0  0.00.00.0   0   0   1   0   0   1 
c0t11d0
0.00.00.00.0  0.0  0.00.00.0   0   0   2   0   0   2 
c3t0d0 
0.00.00.00.0  0.0  0.00.00.0   0   0   2   0   0   2 
c4t0d0 
0.09.00.04.0  0.0  0.00.00.5   0   0   1   0   0   1 
c0t12d0
0.09.00.04.0  0.0  0.00.00.1   0   0   1   0   0   1 
c0t13d0
0.09.00.04.5  0.0  0.00.00.1   0   0   1   0   0   1 
c0t14d0
0.08.00.04.5  0.0  0.00.00.2   0   0   1   0   0   1 
c0t15d0
0.09.00.03.5  0.0  0.00.00.1   0   0   1   0   0   1 
c0t16d0
0.09.00.03.5  0.0  0.00.00.1   0   0   1   0   0   1 
c0t17d0
0.0   20.00.0   56.5  0.0  0.00.00.2   0   0   1   0   0   1 
c2t6d0 
0.0   20.00.0   55.0  0.0  0.00.00.3   0   0   1   0   0   1 
c2t7d0 
0.0   20.00.0   53.5  0.0  0.00.00.2   0   0   1   0   0   1 
c2t8d0 
0.0   20.00.0   53.0  0.0  0.00.00.3   0   0   1   0   0   1 
c2t9d0 
0.0   20.00.0   55.5  0.0  0.00.00.2   0   0   1   0   0   1 
c2t10d0
0.0   20.00.0   55.0  0.0  0.00.00.3   0   0   1   0   0   1 
c2t11d0
0.00.00.00.0  0.0  0.00.00.0   0   0   1   0   0   1 
c2t12d0
0.00.00.00.0  0.0  0.00.00.0   0   0   1   0   0   1 
c2t13d0
 cpu

 us sy wt id

  0 47  0 53

extended device statistics    errors ---

r/sw/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b s/w h/w trn tot 
device 
0.00.00.00.0  0.0  0.00.00.0   0   0   0   0   0   0 
fd0
0.00.00.00.0  0.0  0.00.00.0   0   0   2   0   0   2 
c1t1d0 
0.00.00.00.0  0.0  0.00.00.0   0   0   1   0   0   1 
c0t10d0
0.00.00.00.0  0.0  0.00.00.0   0   0   1   0   0   1 
c0t11d0
0.00.00.00.0  0.0  0.00.00.0   0   0   2   0   0   2 
c3t0d0 
0.00.00.00.0  0.0  0.00.00.0   0   0   2   0   0   2 
c4t0d0 
0.08.00.0   18.5  0.0  0.00.00.2   0   0   1   0   0   1 
c0t12d0
0.08.00.0   18.5  0.0  0.00.00.3   0   0   1   0   0   1 
c0t13d0
0.0   11.00.0   20.5  0.0  0.00.00.3   0   0   1   0   0   1 
c0t14d0
0.0   12.00.0   20.5  0.0  0.00.00.3   0   0   1   0   0   1 
c0t15d0
0.08.00.0   19.0  0.0  0.00.00.2   0   0   1   0   0   1 
c0t16d0
0.08.00.0   18.5  0.0  0.00.00.2   0   0   1   0   0   1 
c0t17d0
0.0   21.00.0   66.0  0.0  0.00.00.4   0   1   1   0   0   1 
c2t6d0 
0.0   21.00.0   66.0  0.0  0.00.00.3   0   0   1   0   0   1 
c2t7d0 
0.0   21.00.0   65.5  0.0  0.00.00.3   0   0   1   0   0   1 
c2t8d0 
0.0   20.00.0   64.0  0.0  0.00.00.4   0   0   1   0   0   1 
c2t9d0 
0.0   21.00.0   65.0  0.0  0.00.00.4   0   0   1   0   0   1 
c2t10d0
0.0   21.00.0   64.0  0.0  0.00.00.3   0   0   1   0   0   1 
c2t11d0
0.00.00.00.0  0.0  0.00.00.0   0   0   1   0   0   1 
c2t12d0
0.00.00.00.0  0.0  0.00.00.0   0   0   1   0   0   1 
c2t13d0
 cpu

 us sy wt id

  0 23  0 77




At other time, drives on controller2  have larger writes (3-10 times) than the 
ones on 
controller1:
extended device statistics    errors ---

r/sw/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b s/w h/w trn tot 
device 
0.00.00.00.0  0.0  0.00.00.0   0   0   0   0   0   0 
fd0
0.00.00.00.0  0.0  0.00.00.0   0   0   2   0   0   2 
c1t1d0 
0.00.00.00.0  0.0  0.00.00

[zfs-discuss] How are you supposed to remove faulted spares from pools?

[Chris Siebenmann]

 We have a situation where all of the spares in a set of pools have
gone into a faulted state and now, apparently, we can't remove them
or otherwise de-fault them. I'm confidant that the underlying disks
are fine, but ZFS seems quite unwilling to do anything with the spares
situation.

(The specific faulted state is 'FAULTED   corrupted data' in
'zpool status' output.)

 Environment: Solaris 10 U6 on x86 hardware. The disks are iSCSI LUNs
from backend storage devices.

 I have tried:
- 'zpool remove': it produces no errors, but doesn't remove anything.
- 'zpool replace  ': it reports that the device is reserved
  as a hot spare.
- 'zpool replace   ': also reports 'device
  is reserved as a hot spare'.
- 'zpool clear': reports that it can't clear errors, the device is
  reserved as a hot spare.

 Because these are iSCSI LUNs, I can actually de-configure them (on the
Solaris side); would that make ZFS change its mind about the situation
and move to a state where I could remove them from the pools?

(Would exporting and then importing the pools make any difference,
especially if the iSCSI LUNs of the spares were removed? These are
production pools, so I can't just try it to see; it would create a
user-visible downtime.)

- cks
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Poor iSCSI performance [SEC=PERSONAL]

[Duncan Groenewald]

All fixed now ...  Backup has been running for maybe a minute or two and has 
backed up over 1GB.

Thanks guys...

Here is the complete command set I used...

Creating the ZFS iSCSI target using COMSTAR.

1.  DO NOT use "ZFS set shareiscsi=on ..."  
2.  MAKE SURE your zpool is upgraded.  Run "zpool upgrade" to see and "zpool 
upgrade storagepool" to upgrade

Commands

01:~# [b]itadm create-target[/b]
Target iqn.1986-03.com.sun:02:7af8d188-b1e8-4d98-fee1-f4da18bbe46f successfully 
created

02:~# [b]itadm list-target -v[/b]
TARGET NAME STATE SESSIONS
iqn.1986-03.com.sun:02:7af8d188-b1e8-4d98-fee1-f4da18bbe46f online 0
alias: -
auth: none (defaults)
targetchapuser: -
targetchapsecret: unset
tpg-tags: default

03:~# [b]zfs create storagepool/backups/iscsi[/b]

04:~# [b]zfs create -V 200g storagepool/backups/iscsi/macbook_dg[/b]

05:~# [b]sbdadm create-lu /dev/zvol/rdsk/storagepool/backups/iscsi/macbook_dg
[/b]
06:~# [b]stmfadm list-lu -v[/b]
LU Name: 600144F827723ED54A95FD4B0002
Operational Status: Online
Provider Name : sbd
Alias : /dev/zvol/rdsk/storagepool/backups/iscsi/macbook_dg
View Entry Count  : 0
Data File : /dev/zvol/rdsk/storagepool/backups/iscsi/macbook_dg
Meta File : not set
Size  : 214748364800
Block Size: 512
Vendor ID : SUN 
Product ID: COMSTAR 
Serial Num: not set
Write Protect : Disabled
Writeback Cache   : Enabled

07:~# [b]stmfadm list-target -v[/b]
Target: iqn.1986-03.com.sun:02:7af8d188-b1e8-4d98-fee1-f4da18bbe46f
Operational Status: Online
Provider Name : iscsit
Alias : -
Sessions  : 0

08:~# [b]stmfadm list-view -l 600144F827723ED54A95FD4B0002[/b]
stmfadm: 600144f827723ed54a95fd4b0002: no views found

09:~# [b]stmfadm add-view 600144F827723ED54A95FD4B0002[/b]

10:~# [b]stmfadm list-view --lu-name 600144F827723ED54A95FD4B0002[/b]
View Entry: 0
Host group   : All
Target group : All
LUN  : 0
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Poor iSCSI performance [SEC=PERSONAL]

[Duncan Groenewald]

Cool - just found the problem.  I had to upgrade the zpool using 
   upgrade zpool storagepool

onwards...
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Poor iSCSI performance [SEC=PERSONAL]

[Duncan Groenewald]

No unfortunately the type does not fix it !!

Still stuck !!
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How to find poor performing disks

[Dave Koelmeyer]

Maybe you can run a Dtrace probe using Chime?

http://blogs.sun.com/observatory/entry/chime

Initial Traces -> Device IO
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Richard Elling]

On Aug 26, 2009, at 1:17 PM, thomas wrote:


Hi Richard,



So you have to wait for the sd (or other) driver to
timeout the request. By
default, this is on the order of minutes. Meanwhile,
ZFS is patiently awaiting a status on the request. For
enterprise class drives, there is a limited number
of retries on the disk before it reports an error.
You can expect responses of success in the order of
10 seconds or less. After the error is detected, ZFS
can do something about it.

All of this can be tuned, of course.  Sometimes that
tuning is ok by default, sometimes not. Until recently, the
biggest gripes were against the iscsi client which had a
hardwired 3 minute error detection. For current
builds you can tune these things without recompiling.
 -- richard



So are you suggesting that tuning the sd driver's settings to  
timeout sooner if using

a consumer class drive wouldn't be wise for perhaps other reasons?


Unfortunately, it requires skill and expertise :-(. Getting it
wrong can lead to an unstable system. For this reason, the
main users of such tuning are the large integrators or IHVs.

Note: this sort of thing often does not have an immediate or
obvious impact. But it can have an impact when a failure or
unanticipated event occurs. In other words, when the going
gets tough, serviceability can be negatively impacted. Murphy's
law implies that will happen at the worst possible moment.
 -- richard

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] problem with zfs

[serge goyette]

Hum
i know the recommended and the update patches bundles
according to the readme

Patch 139555-08 is the kernel patch associated with the Solaris 10 5/09 release 
(Solaris 10 Update 7).

so i believe i'm up to date

i understand i'm a bit vague but i cannot provide any Zpool or zfs output 
until i reboot.

all the pools were created after the OS update
just doing import, which is working .
than doing a mount on the FS. the mountpoint is set to legacy
so something like mount -F zfs  POOL/FS /MOUNTPOINT

it does work but from time to time (intermittant) the command hang there
then nightmare starts

am i alone to experiment this ?
very hard to troubleshoot if i cannot execute any commands
anyway i opened a case at Sun and not planning to reboot until they call me back
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How to find poor performing disks

[Scott Lawson]

Also you may wish to look at the output of 'iostat -xnce 1' as well.

You can post those to the list if you have a specific problem.

You want to be looking for error counts increasing and specifically 'asvc_t'
for the service times on the disks. I higher number for asvc_t  may help to
isolate poorly performing individual disks.



Scott Meilicke wrote:

You can try:

zpool iostat pool_name -v 1

This will show you IO on each vdev at one second intervals. Perhaps you will 
see different IO behavior on any suspect drive.

-Scott
  



___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] problem with zfs

[Scott Lawson]

serge goyette wrote:

actually i did apply the latest recommended patches
  

Recommended patches and upgrade clusters are different by the way.

10_Recommended != Upgrade Cluster that. Upgrade cluster will upgrade
the system to a effectively the Solaris Release  that the upgrade cluster
is minus any new features that arrived in the newer OS release.

SunOS VL-MO-ZMR01 5.10 Generic_139555-08 sun4v sparc SUNW,SPARC-Enterprise-T5120

but still 


perhaps you are not doing much import - export
because when i do not do, i do not experience much problem
but when doing it, outch ...
  
Sure I import and export pools. But generally this is for moving the 
pool to another

system.

But I think we would need more information about the pool and
it's file systems to be able to help you. Specifically maybe the output of
'zpool history' and 'zfs list' for starters. This will at least allow 
some specific
data to try and help resolve your issues. The question as it stands is 
pretty

generic.

Have you upgraded your pools after the patches as well?

'zpool upgrade' and 'zfs upgrade' ?

a reboot will solve until next time

-sego-
  


-

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[thomas]

Hi Richard,


> So you have to wait for the sd (or other) driver to
> timeout the request. By
> default, this is on the order of minutes. Meanwhile,
> ZFS is patiently awaiting a status on the request. For
> enterprise class drives, there is a limited number
> of retries on the disk before it reports an error.
> You can expect responses of success in the order of
> 10 seconds or less. After the error is detected, ZFS
> can do something about it.
> 
> All of this can be tuned, of course.  Sometimes that
> tuning is ok by default, sometimes not. Until recently, the
> biggest gripes were against the iscsi client which had a
> hardwired 3 minute error detection. For current  
> builds you can tune these things without recompiling.
>   -- richard


So are you suggesting that tuning the sd driver's settings to timeout sooner if 
using
a consumer class drive wouldn't be wise for perhaps other reasons?
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] problem with zfs

[serge goyette]

actually i did apply the latest recommended patches

SunOS VL-MO-ZMR01 5.10 Generic_139555-08 sun4v sparc SUNW,SPARC-Enterprise-T5120

but still 

perhaps you are not doing much import - export
because when i do not do, i do not experience much problem
but when doing it, outch ...

a reboot will solve until next time

-sego-
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] problem with zfs

[Scott Lawson]

The latest official Solaris 10 is actually 05/09. There are update patch 
bundles available
on Sunsolve for free download that will take you to 05/09. It may well 
be worth applying
these to see if they remedy the problem for you. They certainly allow 
you to bring ZFS up to version
10 from recollection. I have upgraded 30 plus systems with these and 
haven't experienced any

issues. (both SPARC and x86)

http://sunsolve.sun.com/pdownload.do?target=10_sparc_0509_patchbundle_part1.zip
http://sunsolve.sun.com/pdownload.do?target=10_sparc_0509_patchbundle_part2.zip
http://sunsolve.sun.com/pdownload.do?target=10_sparc_0509_patchbundle_part3.zip
http://sunsolve.sun.com/pdownload.do?target=10_sparc_0509_patchbundle_part4.zip


serge goyette wrote:

for release sorry i meant

 Solaris 10 10/08 s10s_u6wos_07b SPARC
   Copyright 2008 Sun Microsystems, Inc.  All Rights Reserved.
Use is subject to license terms.
Assembled 27 October 2008
  


--
___


Scott Lawson
Systems Architect
Manukau Institute of Technology
Information Communication Technology Services Private Bag 94006 Manukau
City Auckland New Zealand

Phone  : +64 09 968 7611
Fax: +64 09 968 7641
Mobile : +64 27 568 7611

mailto:sc...@manukau.ac.nz

http://www.manukau.ac.nz




perl -e 'print
$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10);'

 


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] problem with zfs

[serge goyette]

for release sorry i meant

 Solaris 10 10/08 s10s_u6wos_07b SPARC
   Copyright 2008 Sun Microsystems, Inc.  All Rights Reserved.
Use is subject to license terms.
Assembled 27 October 2008
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How to find poor performing disks

[Scott Meilicke]

You can try:

zpool iostat pool_name -v 1

This will show you IO on each vdev at one second intervals. Perhaps you will 
see different IO behavior on any suspect drive.

-Scott
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[zfs-discuss] problem with zfs

[serge goyette]

HI 

i'm using latest official solaris version 10/06 = patches and 
i'm getting problem

after import of zpool  sometime when i do 
mount -F zfs zpool/fs  /mountpoint

the command freeze
and after this these is no way i can umount, destroy or anything
all commands will hung

even rebooting appears to be an issue


-sego-
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Tim Cook]

On Wed, Aug 26, 2009 at 11:45 AM, Neal Pollack  wrote:

>
> Luck or "design/usage" ?
> Let me explain;   I've also had many drives fail over the last 25
> years of working on computers, I.T., engineering, manufacturing,
> and building my own PCs.
>
> Drive life can be directly affected by heat.  Many home tower designs,
> until the last year or two, had no cooling fans or air flow where
> the drives mount.  I'd say over 80% of desktop "average" PCs do
> not have any cooling or air flow for the drive.
> (I've replaced many many for friends).
> [HP small form factor desktops are the worst offenders
>  in what I jokingly call "zero cooling design" :-)
> Just look at the quantity of refurbished ones offered for sale]
>
> Once I started adding cooling fans for my drives in my own
> workstations I build, the rate of drive failures went
> down by a lot.  The drive life went up by a lot.
>
> You can still have random failures for a dozen reasons, but
> heat is one of the big killers.  I did some experiments over
> the last 5 years and found that ANY amount of air flow makes
> a big difference.  If you run a 12 volt fan at 7 volts by
> connecting it's little red and black wires across the outside
> of a disk drive connecter (red and orange wires, 12 and 5 volt, difference
> is 7), then the fan is silent, moves a small flow of air, and drops
> the disk drive temperature by a lot.
> [Translation:  It can be as quiet as a dell, but twice as good
> since you built it :-) ]
>
> That said, there are some garbage disk drive designs on the market.
> But if a lot of yours fail early, close to warranty, they might
> be getting abused or run near the max design temperature?
>
> Neal
>

I've always cooled my drives.  I just blame it on MAXTOR having horrible
designs.

Funny, everyone bagged on the 75GXP's from IBM, but I had a pair that I
bought when they first came out, used them for 5 years, then sold them to a
friend who got at least another 3 years out of them (heck, he might still be
using them for all I know).  Those maxtor's weren't worth the packaging they
came in.  I wasn't sad to see them bought up.

--Tim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Neal Pollack]

On 08/25/09 10:46 PM, Tim Cook wrote:
On Wed, Aug 26, 2009 at 12:22 AM, thomas > wrote:


 > I'll admit, I was cheap at first and my
 > fileserver right now is consumer drives.  You
 > can bet all my future purchases will be of the enterprise grade.
 And
 > guess what... none of the drives in my array are less than 5
years old, so even
 > if they did die, and I had bought the enterprise versions, they'd be
 > covered.

Anything particular happen that made you change your mind? I started
with
"enterprise grade" because of similar information discussed in this
thread.. but I've
also been wondering how zfs holds up with consumer level drives and
if I could save
money by using them in the future. I guess I'm looking for horror
stories that can be
attributed to them? ;)



When it comes to my ZFS project, I am currently lacking horror stories.  
When it comes to "what the hell, this drive literally failed a week 
after the warranty was up", I unfortunately PERSONALLY have 3 examples.  
I'm guessing (hoping) it's just bad luck.  



Luck or "design/usage" ?
Let me explain;   I've also had many drives fail over the last 25
years of working on computers, I.T., engineering, manufacturing,
and building my own PCs.

Drive life can be directly affected by heat.  Many home tower designs,
until the last year or two, had no cooling fans or air flow where
the drives mount.  I'd say over 80% of desktop "average" PCs do
not have any cooling or air flow for the drive.
(I've replaced many many for friends).
[HP small form factor desktops are the worst offenders
 in what I jokingly call "zero cooling design" :-)
Just look at the quantity of refurbished ones offered for sale]

Once I started adding cooling fans for my drives in my own
workstations I build, the rate of drive failures went
down by a lot.  The drive life went up by a lot.

You can still have random failures for a dozen reasons, but
heat is one of the big killers.  I did some experiments over
the last 5 years and found that ANY amount of air flow makes
a big difference.  If you run a 12 volt fan at 7 volts by
connecting it's little red and black wires across the outside
of a disk drive connecter (red and orange wires, 12 and 5 volt, difference
is 7), then the fan is silent, moves a small flow of air, and drops
the disk drive temperature by a lot.
[Translation:  It can be as quiet as a dell, but twice as good
since you built it :-) ]

That said, there are some garbage disk drive designs on the market.
But if a lot of yours fail early, close to warranty, they might
be getting abused or run near the max design temperature?

Neal


Perhaps the luck wasn't SO
bad though, as I had backups of all of those (proof, you should never 
rely on a single drive to last up to,or beyond its warranty period.


--Tim




___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Richard Elling]

On Aug 25, 2009, at 9:38 PM, Tristan Ball wrote:

What I’m worried about that time period where the pool is  
resilvering to the hot spare. For example: one half of a mirror has  
failed completely, and the mirror is being rebuilt onto the spare –  
if I get a read error from the remaining half of the mirror, then  
I’ve lost data. If the RE drives return’s an error for a request  
that a consumer drive would have (eventually) returned, then in this  
specific case I would have been better off with the consumer drive.


The difference is the error detection time. In general, you'd like  
errors to be
detected quickly.  The beef with the consumer drives and the Solaris +  
ZFS
architecture is that the drives do not return on error, they just keep  
trying.
So you have to wait for the sd (or other) driver to timeout the  
request. By
default, this is on the order of minutes. Meanwhile, ZFS is patiently  
awaiting
a status on the request. For enterprise class drives, there is a  
limited number
of retries on the disk before it reports an error. You can expect  
responses of
success in the order of 10 seconds or less. After the error is  
detected, ZFS

can do something about it.

All of this can be tuned, of course.  Sometimes that tuning is ok by  
default,

sometimes not. Until recently, the biggest gripes were against the iscsi
client which had a hardwired 3 minute error detection. For current  
builds

you can tune these things without recompiling.
 -- richard

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Adam Sherman]

But the real question is whether the "enterprise" drives would have  
avoided your problem.


A.

--
Adam Sherman
+1.613.797.6819

On 2009-08-26, at 11:38, Troels Nørgaard Nielsen  wrote:


Hi Tim Cook.

If I was building my own system again, I would prefer not to go with  
consumer harddrives.
I had a raidz pool containing eight drives on a snv108 system, after  
rebooting, four of the eight drives was so broken they could not be  
seen by format, let alone the zpool they belonged to.


This was with Samsung HD103UJ revision 1112 and 1113 disks. No  
matter what kind of hotspares, raidz2 or 3-way mirror would have  
saved me, so it was RMA the drives, buy some new and restore from  
backup. The controller was LSI1068E - A cheap USB-to-SATA adapter  
could see them, but with masive stalls and errors.
These disks was at the moment the cheapest 1 TB disks avaliable, I  
understand why now.


But I'm still stuck with 6 of them in my system ;-(

Best regards
Troels Nørgaard

Den 26/08/2009 kl. 07.46 skrev Tim Cook:

On Wed, Aug 26, 2009 at 12:22 AM, thomas   
wrote:

> I'll admit, I was cheap at first and my
> fileserver right now is consumer drives.  You
> can bet all my future purchases will be of the enterprise grade.  
 And
> guess what... none of the drives in my array are less than 5  
years old, so even
> if they did die, and I had bought the enterprise versions, they'd  
be

> covered.

Anything particular happen that made you change your mind? I  
started with
"enterprise grade" because of similar information discussed in this  
thread.. but I've
also been wondering how zfs holds up with consumer level drives and  
if I could save
money by using them in the future. I guess I'm looking for horror  
stories that can be

attributed to them? ;)


When it comes to my ZFS project, I am currently lacking horror  
stories.  When it comes to "what the hell, this drive literally  
failed a week after the warranty was up", I unfortunately  
PERSONALLY have 3 examples.  I'm guessing (hoping) it's just bad  
luck.  Perhaps the luck wasn't SO bad though, as I had backups of  
all of those (proof, you should never rely on a single drive to  
last up to,or beyond its warranty period.


--Tim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Joseph L. Casale]

>If I was building my own system again, I would prefer not to go with consumer 
>harddrives.
>I had a raidz pool containing eight drives on a snv108 system, after 
>rebooting, four of
>the eight drives was so broken they could not be seen by format, let alone the 
>zpool they
>belonged to.
>
>This was with Samsung HD103UJ revision 1112 and 1113 disks. No matter what 
>kind of hotspares,
>raidz2 or 3-way mirror would have saved me, so it was RMA the drives, buy some 
>new and restore
>from backup. The controller was LSI1068E - A cheap USB-to-SATA adapter could 
>see them, but with
>masive stalls and errors. These disks was at the moment the cheapest 1 TB 
>disks avaliable, I
>understand why now.

I can attest to the same experience with very nearly the same hardware. My next 
non critical system
will not use consumer HD's either. The stalling issue seemed to vary with 
different hardware but
left my chasing my tail endlessly. I am sure Seagate has a bulletin with my 
name on it...

jlc
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[zfs-discuss] How to find poor performing disks

[Simon Gao]

Hi,

I'd appreciate if anyone can point me how to identify poor performing disks 
that might have dragged down performance of the pool. Also the system logged 
following error about one of the drives. Does it show the disk was having 
problem?

Aug 17 13:45:56 zfs1.domain.com scsi: [ID 107833 kern.warning] WARNING: 
/p...@0,0/pci8086,2...@6/pci1000,3...@0 (mpt1):
Aug 17 13:45:56 zfs1.domain.com  Disconnected command timeout for Target 10
Aug 17 13:45:56 zfs1.domain.com scsi: [ID 365881 kern.info] 
/p...@0,0/pci8086,2...@6/pci1000,3...@0 (mpt1):
Aug 17 13:45:56 zfs1.domain.com  Log info 3114 received for target 10.
Aug 17 13:45:56 zfs1.domain.com  scsi_status=0, ioc_status=8048, scsi_state=c   
 
Aug 17 13:45:56 zfs1.domain.com scsi: [ID 107833 kern.warning] WARNING: 
/p...@0,0/pci8086,2...@6/pci1000,3...@0/s...@a,0 (sd15):
Aug 17 13:45:56 zfs1.domain.com  SCSI transport failed: reason 'reset': 
retrying command 
Aug 17 13:45:59 zfs1.domain.com scsi: [ID 107833 kern.warning] WARNING: 
/p...@0,0/pci8086,2...@6/pci1000,3...@0/s...@a,0 (sd15):
Aug 17 13:45:59 zfs1.domain.com  Error for Command: read(10)
Error Level: Retryable
Aug 17 13:45:59 zfs1.domain.com scsi: [ID 107833 kern.notice]Requested 
Block: 715872929 Error Block: 715872929
Aug 17 13:45:59 zfs1.domain.com scsi: [ID 107833 kern.notice]Vendor: ATA
Serial Number:  WD-WCAP
Aug 17 13:45:59 zfs1.domain.com scsi: [ID 107833 kern.notice]Sense Key: 
Unit Attention
Aug 17 13:45:59 zfs1.domain.com scsi: [ID 107833 kern.notice]ASC: 0x29 
(power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Troels Nørgaard Nielsen]

Hi Tim Cook.

If I was building my own system again, I would prefer not to go with  
consumer harddrives.
I had a raidz pool containing eight drives on a snv108 system, after  
rebooting, four of the eight drives was so broken they could not be  
seen by format, let alone the zpool they belonged to.


This was with Samsung HD103UJ revision 1112 and 1113 disks. No matter  
what kind of hotspares, raidz2 or 3-way mirror would have saved me, so  
it was RMA the drives, buy some new and restore from backup. The  
controller was LSI1068E - A cheap USB-to-SATA adapter could see them,  
but with masive stalls and errors.
These disks was at the moment the cheapest 1 TB disks avaliable, I  
understand why now.


But I'm still stuck with 6 of them in my system ;-(

Best regards
Troels Nørgaard

Den 26/08/2009 kl. 07.46 skrev Tim Cook:


On Wed, Aug 26, 2009 at 12:22 AM, thomas  wrote:
> I'll admit, I was cheap at first and my
> fileserver right now is consumer drives.  You
> can bet all my future purchases will be of the enterprise grade.  
 And
> guess what... none of the drives in my array are less than 5 years  
old, so even

> if they did die, and I had bought the enterprise versions, they'd be
> covered.

Anything particular happen that made you change your mind? I started  
with
"enterprise grade" because of similar information discussed in this  
thread.. but I've
also been wondering how zfs holds up with consumer level drives and  
if I could save
money by using them in the future. I guess I'm looking for horror  
stories that can be

attributed to them? ;)


When it comes to my ZFS project, I am currently lacking horror  
stories.  When it comes to "what the hell, this drive literally  
failed a week after the warranty was up", I unfortunately PERSONALLY  
have 3 examples.  I'm guessing (hoping) it's just bad luck.  Perhaps  
the luck wasn't SO bad though, as I had backups of all of those  
(proof, you should never rely on a single drive to last up to,or  
beyond its warranty period.


--Tim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss




smime.p7s
Description: S/MIME cryptographic signature
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Bob Friesenhahn]

On Wed, 26 Aug 2009, Tristan Ball wrote:


Complete disk failures are comparatively rare, while media or transient
errors are far more common. As a media I/O or transient error on the


It seems that this assumption is is not always the case.  The 
expensive small-capacity SCSI/SAS enterprise drives rarely experience 
media errors so total drive failure becomes a larger factor.  Large 
capacity SATA drives tend to report many more media failures and the 
whole drive failure rate is perhaps not much worse than enterprise 
SCSI/SAS drives.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[zfs-discuss] Save modification to liveUSB

[k1rld5]

Hello all,

I used liveUSB Creator to create a OpenSolaris LiveUSB. It's booting fast and 
easy.

Mainly I'll need it to deploy packages (use it for jumpstart or so). I'll need 
to stick on a USB stick (16GB) and be able to reach it from the network.

So far it is good, but now the issue is that if I install applications on it, 
then the next boot will reset everything.

I wanted to know if there was a way to make it persistent (even if it is by 
modifying everything by hand).

I did already try to install opensolaris on the USB Stick, but no success, each 
time I can boot when done (no PBR sig message). I also tried a belenix version 
with usbdump, no chance neither. I'd like to try what I found on 
osholes.blogspot.com, but this requires a install on USB

I think that a liveUSB is the same as a liveCD, with the advantage that the USB 
is writable. So I should be able to install more application onto the CD part 
of it (like in /.cdrom)

Or better, it could be good to be able to add a persistent file like for ubuntu 
or fedora (casper-rw)
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Thomas Burgess]

I've been running ZFS under FreeBSD which is experimental, and i've had
nothing but great luckso i guess it depeneds on a number of things.  I
went with FreeBSD because the hardware i had wasn't supported in
solarisi expected problems but honestly, it's been rock solidit's
survived all kinds of things...power failures, a drive failure, partial
drive failure and a power supply dying...
I'm running 3 raidz1 VDEVS all with 1TB drives.   I've asked in the FreeBSD
forums for "horror stories" because i see SO many on this thread but so far,
noone has has ANYTHING bad to say about ZFS in FreeBSD.

I'm sure this is somewhat due to the fact that it's not as popular in
FreeBSD yet but i still expected to hear SOME horror storiesFrom my
experience and the experiences of the other people in the FreeBSD forums
it's been great running on both "non-sun" hardware AND software.


On Wed, Aug 26, 2009 at 3:06 AM, Ross  wrote:

> Here's one horror story of mine - ZFS taking over 20 minutes to flag a
> drive as faulty, with the entire pool responding so slowly during those 20
> minutes that it crashed six virtual machines running off the pool:
> http://www.opensolaris.org/jive/thread.jspa?messageID=369265񚉱
>
> There are some performance tweaks mentioned in that thread, but I haven't
> been able to test their effectiveness yet, and I'm still concerned.  A pool
> consisting of nothing but three way mirrors should not even break a sweat
> when faced with a single drive problem.
>
> When you're implementing what's sold as the most advanced file system ever,
> billed as good enough to obsolete raid controllers, you don't expect to be
> doing manual tweaks just so it can cope with a drive failure without hanging
> the entire pool.
>
> ZFS has its benefits, but if you're not running it on Sun hardware you need
> to do a *lot* of homework.
> --
> This message posted from opensolaris.org
> ___
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zpool import hangs with indefinite writes

[Victor Latushkin]

On 21.08.09 14:52, No Guarantees wrote:

Every time I attempt to import a particular RAID-Z pool, my system hangs.
Specifically, if I open up a gnome terminal and input '$ pfexec import zpool
mypool', the process will never complete and I will return to the prompt.  If
I open up another terminal, I can input a 'zpool status" and see that the
pool has been imported, but the directory has not been mounted.


This suggests that import is partially done, it is just unable to perform some 
final stage of the process, so 'zpool import' never returns.


So you need to do something like this to see where 'zpool import' is stuck.

1. Find PID of the hanging 'zpool import', e.g. with 'ps -ef | grep zpool'

2. Substitute PID with actual number in the below command

echo "0tPID::pid2proc|::walk thread|::findstack -v" | mdb -k

3. Do

echo "::spa" | mdb -k

4. Find address of your pool in the output of stage 3 and replace ADDR with it 
in the below command (it is single line):


echo "ADDR::print spa_t spa_dsl_pool->dp_tx.tx_sync_thread|::findstack -v" | 
mdb -k

5. Run command in step 4 several times.

This could be the first step. Another option may be to force a crash dump.


 In other
words, there is no /mypool in the tree.  If I issue a 'zpool iostat 1' I can
see that there are constant writes to the pool, but NO reads.  If I halt the
zpool import,


What do you mean by halt here? Are you able to interrupt 'zpool import' with 
CTRL-C?


and then do a 'zpool scrub', it will complete with no errors
after about 12-17 hours (it's a 5TB pool).


That sound promising.

Does 'zfs list' provide any output?

Apparently as you have 5TB of data there, it worked fine some time ago. What 
happened to the pool before this issue was noticed?


regards,
victor


 I have looked through this forum
and found many examples where people can't import due to hardware failure and
lack of redundancy, but none where they had a redundant setup, everything
appears okay, and they STILL can't import.  I can export the pool without any
problems.  I need to do this before rebooting, otherwise it hangs on reboot, 
probably while trying to import the pool.  I've looked around for

troubleshooting info, but the only thing that gives me any hint of what is
wrong is a core dump after issuing a 'zdb -v mypool'.  It fails with
"Assertion failed: object_count == usedobjs (0x7 == 0x6), file ../zdb.c, line
1215.  Any suggestions?





_
Windows Live: Keep your friends up to date with what you do online.
http://windowslive.com/Campaign/SocialNetworking?ocid=PID23285::T:WLMTAGL:ON:WL:en-US:SI_SB_online:082009
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Michael Herf]

I'm using the Caviar Green drives in a 5-disk config.

I downloaded the WDTLER utility and set all the drives to have a 7-second 
timeout, like the RE series have.

WDTLER boots a small DOS app and you have to hit a key for each drive to 
adjust. So this might take time for a large raidz2.
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zpool import hangs with indefinite writes

[noguaran]

Okay, I'm trying to do whatever I can NONDESTRUCTIVELY to fix this.  I have 
almost 5TB of data that I can't afford to lose (baby pictures and videos, 
etc.).  Since no one has seen this problem before, maybe someone can tell me 
what I need to do to make a backup of what I have now so I can try other 
methods to recover this data.  Just about everything I do starts writing to 
these drives, and that worries me.  For example:  zfs list - writes, zfs 
volinit - writes.  I am hoping that I have not already ruined the existing data 
on these drives, but I do not know enough about ZFS to troubleshoot or test.  
I'm a little bit frustrated about this because I think I have everything I 
need, but I still can't access anything:  64-bit - check, ECC RAM - check, 
redundancy (RAID-Z) - check.  I don't know if I'm not following the proper 
protocol for posting, and that is why I am not getting any responses, or what.  
At this point, I'm open to any suggestions (right or wrong).  If the only way 
 I'll get any help is to pay for OpenSolaris support, let me know.
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[zfs-discuss] zpool scrub started resilver, not scrub

[Albert Chin]

# cat /etc/release
  Solaris Express Community Edition snv_105 X86
   Copyright 2008 Sun Microsystems, Inc.  All Rights Reserved.
Use is subject to license terms.
   Assembled 15 December 2008
# zpool status tww
  pool: tww
 state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: resilver completed after 6h15m with 27885 errors on Wed Aug 26 07:18:03 
2009
config:

NAME   STATE READ WRITE CKSUM
twwONLINE   0 0 54.5K
  raidz2   ONLINE   0 0 0
c6t600A0B800029996605964668CB39d0  ONLINE   0 0 0
c6t600A0B8000299CCC06C84744C892d0  ONLINE   0 0 0
c6t600A0B8000299CCC05B44668CC6Ad0  ONLINE   0 0 0
c6t600A0B800029996605A44668CC3Fd0  ONLINE   0 0 0
c6t600A0B8000299CCC05BA4668CD2Ed0  ONLINE   0 0 0
c6t600A0B800029996605AA4668CDB1d0  ONLINE   0 0 0
c6t600A0B8000299966073547C5CED9d0  ONLINE   0 0 0
  raidz2   ONLINE   0 0 0
c6t600A0B800029996605B04668F17Dd0  ONLINE   0 0 0
c6t600A0B8000299CCC099E4A400B94d0  ONLINE   0 0 0
c6t600A0B800029996605B64668F26Fd0  ONLINE   0 0 0
c6t600A0B8000299CCC05CC4668F30Ed0  ONLINE   0 0 0
c6t600A0B800029996605BC4668F305d0  ONLINE   0 0 0
c6t600A0B8000299CCC099B4A400A9Cd0  ONLINE   0 0 0
c6t600A0B800029996605C24668F39Bd0  ONLINE   0 0 0
  raidz2   ONLINE   0 0  109K
c6t600A0B8000299CCC0A154A89E426d0  ONLINE   0 0 0
c6t600A0B800029996609F74A89E1A5d0  ONLINE   0 018  
2.50K resilvered
c6t600A0B8000299CCC0A174A89E520d0  ONLINE   0 039  
4.50K resilvered
c6t600A0B800029996609F94A89E24Bd0  ONLINE   0 0   486  
75K resilvered
c6t600A0B80002999660A454A93CEDBd0  ONLINE   0 0 0  
2.55G resilvered
c6t600A0B8000299CCC0A0C4A89DDE8d0  ONLINE   0 034  
2K resilvered
c6t600A0B800029996609F04A89DB1Bd0  ONLINE   0 0   173  
18K resilvered
spares
  c6t600A0B8000299CCC05D84668F448d0AVAIL   
  c6t600A0B800029996605C84668F461d0AVAIL   

errors: 27885 data errors, use '-v' for a list

# zpool scrub tww
# zpool status tww
  pool: tww
 state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: resilver in progress for 0h11m, 2.82% done, 6h21m to go
config:
...

So, why is a resilver in progress when I asked for a scrub?

-- 
albert chin (ch...@thewrittenword.com)
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Using consumer drives in a zraid2

[Ross]

Here's one horror story of mine - ZFS taking over 20 minutes to flag a drive as 
faulty, with the entire pool responding so slowly during those 20 minutes that 
it crashed six virtual machines running off the pool:
http://www.opensolaris.org/jive/thread.jspa?messageID=369265񚉱

There are some performance tweaks mentioned in that thread, but I haven't been 
able to test their effectiveness yet, and I'm still concerned.  A pool 
consisting of nothing but three way mirrors should not even break a sweat when 
faced with a single drive problem.

When you're implementing what's sold as the most advanced file system ever, 
billed as good enough to obsolete raid controllers, you don't expect to be 
doing manual tweaks just so it can cope with a drive failure without hanging 
the entire pool.

ZFS has its benefits, but if you're not running it on Sun hardware you need to 
do a *lot* of homework.
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss