[zfs-discuss] (Meta)Data corrupted on multiple triple mirrors
Hello.
3 month old pool hosted across two SAS JBODs w/ SATA disks on an x4170m2
running Solaris 11 SRU6.6, ran a scrub and I got this:
pool: zdata
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scan: scrub repaired 0 in 19h51m with 1 errors on Fri Nov 9 18:54:22 2012
config:
NAME STATE READ WRITE CKSUM
zdata ONLINE 0 0 1
mirror-0 ONLINE 0 0 2
c3t1d0 ONLINE 0 0 2
c3t11d0ONLINE 0 0 2
c3t0d0 ONLINE 0 0 2
mirror-1 ONLINE 0 0 2
c3t4d0 ONLINE 0 0 2
c3t14d0ONLINE 0 0 2
c3t3d0 ONLINE 0 0 2
mirror-2 ONLINE 0 0 2
c3t7d0 ONLINE 0 0 2
c3t17d0ONLINE 0 0 2
c3t6d0 ONLINE 0 0 2
logs
c0t5E83A97F1471E0A4d0s0 ONLINE 0 0 0
cache
c0t01507A51E36B3C08d0s0 ONLINE 0 0 0
spares
c3t10d0 AVAIL
errors: Permanent errors have been detected in the following files:
:<0x132c1f>
How did this happen and how screwed are we? I've sent a bug report to
Oracle too..
Log is an OCZ Vertex2EX (slc&supercap), cache is an Intel 510.
Host has been rebooted 3 times after the pool was created, all the same
day as creation.
/Tomas
--
Tomas Forsman, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] cannot replace X with Y: devices have different sector alignment
On 2012-11-10 17:16, Jan Owoc wrote: Any other ideas short of block pointer rewrite? A few... one is an idea of what could be the cause: AFAIK the ashift value is not so much per-pool as per-toplevel-vdev. If the pool started as a set of the 512b drives and was then expanded to include sets of 4K drives, this mixed ashift could happen... It might be possible to override the ashift value with sd.conf and fool the OS into using 512b sectors over a 4KB native disk (this is mostly used the other way around, though - to enforce 4KB sectors on 4KB native drives that emulate 512b sectors). This might work, and earlier posters on the list saw no evidence to say that 512b emulation is inherently evil and unreliable (modulo firmware/hardware errors that can be anywhere anyway), but this would likely make the disk slower on random writes. Also, I am not sure how the 4KB-native HDD would process partial overwrites of a 4KB sector with 512b pieces of data - would other bytes remain intact or not?.. Before trying to fool a production system this way, if at all, I believe some stress-tests with small blocks are due on some other system. My 2c, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] cannot replace X with Y: devices have different sector alignment
On Sat, Nov 10, 2012 at 9:04 AM, Tim Cook wrote: > On Sat, Nov 10, 2012 at 9:59 AM, Jan Owoc wrote: >> Sorry... my question was partly answered by Jim Klimov on this list: >> http://openindiana.org/pipermail/openindiana-discuss/2012-June/008546.html >> >> Apparently the currently-suggested way (at least in OpenIndiana) is to: >> 1) create a zpool on the 4k-native drive >> 2) zfs send | zfs receive the data >> 3) mirror back onto the non-4k drive > > That would absolutely work, but it's not really a fix for this situation. > For OP to do this he'd need 42 new drives (or at least enough drives to > provide the same capacity as what he's using) to mirror to and then mirror > back. The only way this is happening for most people is if they only have a > very small pool, and have the ability to add an equal amount of storage to > dump to. Probably not a big deal if you've only got a handful of drives, or > if the drives you have are small and you can take downtime. Likely > impossible for OP with 42 large drives. Yes, you are right. I missed the fact that this mirror is part of a very large pool, so zfs send | zfs receive isn't exactly an option. Any other ideas short of block pointer rewrite? Jan ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] cannot replace X with Y: devices have different sector alignment
On Sat, Nov 10, 2012 at 9:59 AM, Jan Owoc wrote: > On Sat, Nov 10, 2012 at 8:48 AM, Jan Owoc wrote: > > On Sat, Nov 10, 2012 at 8:14 AM, Trond Michelsen > wrote: > >> When I try to replace the old drive, I get this error: > >> > >> # zpool replace tank c4t5000C5002AA2F8D6d0 c4t5000C5004DE863F2d0 > >> cannot replace c4t5000C5002AA2F8D6d0 with c4t5000C5004DE863F2d0: > >> devices have different sector alignment > >> > >> > >> How can I replace the drive without migrating all the data to a > >> different pool? It is possible, I hope? > > > > I had the same problem. I tried copying the partition layout and some > > other stuff but without success. I ended up having to recreate the > > pool and now have a non-mirrored root fs. > > > > If anyone has figured out how to mirror drives after getting the > > message about sector alignment, please let the list know :-). > > Sorry... my question was partly answered by Jim Klimov on this list: > http://openindiana.org/pipermail/openindiana-discuss/2012-June/008546.html > > Apparently the currently-suggested way (at least in OpenIndiana) is to: > 1) create a zpool on the 4k-native drive > 2) zfs send | zfs receive the data > 3) mirror back onto the non-4k drive > > I can't test it at the moment on my setup - has anyone tested this to work? > > Jan > > That would absolutely work, but it's not really a fix for this situation. For OP to do this he'd need 42 new drives (or at least enough drives to provide the same capacity as what he's using) to mirror to and then mirror back. The only way this is happening for most people is if they only have a very small pool, and have the ability to add an equal amount of storage to dump to. Probably not a big deal if you've only got a handful of drives, or if the drives you have are small and you can take downtime. Likely impossible for OP with 42 large drives. --Tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] cannot replace X with Y: devices have different sector alignment
On Sat, Nov 10, 2012 at 9:48 AM, Jan Owoc wrote: > On Sat, Nov 10, 2012 at 8:14 AM, Trond Michelsen > wrote: > > When I try to replace the old drive, I get this error: > > > > # zpool replace tank c4t5000C5002AA2F8D6d0 c4t5000C5004DE863F2d0 > > cannot replace c4t5000C5002AA2F8D6d0 with c4t5000C5004DE863F2d0: > > devices have different sector alignment > > > > > > How can I replace the drive without migrating all the data to a > > different pool? It is possible, I hope? > > I had the same problem. I tried copying the partition layout and some > other stuff but without success. I ended up having to recreate the > pool and now have a non-mirrored root fs. > > If anyone has figured out how to mirror drives after getting the > message about sector alignment, please let the list know :-). > > Jan > Not happening with anything that exists today. The only way this would be possible is with bp_rewrite which would allow you to evacuate a vdev (whether it be for a situation like this, or just to shrink a pool). What you're trying to do is write a block for block copy to a disk that's made up of a different block structure. Not happening. *insert everyone saying they want bp_rewrite and the guys who have the skills to do so saying their enterprise customers have other needs* --Tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] cannot replace X with Y: devices have different sector alignment
On Sat, Nov 10, 2012 at 8:48 AM, Jan Owoc wrote: > On Sat, Nov 10, 2012 at 8:14 AM, Trond Michelsen wrote: >> When I try to replace the old drive, I get this error: >> >> # zpool replace tank c4t5000C5002AA2F8D6d0 c4t5000C5004DE863F2d0 >> cannot replace c4t5000C5002AA2F8D6d0 with c4t5000C5004DE863F2d0: >> devices have different sector alignment >> >> >> How can I replace the drive without migrating all the data to a >> different pool? It is possible, I hope? > > I had the same problem. I tried copying the partition layout and some > other stuff but without success. I ended up having to recreate the > pool and now have a non-mirrored root fs. > > If anyone has figured out how to mirror drives after getting the > message about sector alignment, please let the list know :-). Sorry... my question was partly answered by Jim Klimov on this list: http://openindiana.org/pipermail/openindiana-discuss/2012-June/008546.html Apparently the currently-suggested way (at least in OpenIndiana) is to: 1) create a zpool on the 4k-native drive 2) zfs send | zfs receive the data 3) mirror back onto the non-4k drive I can't test it at the moment on my setup - has anyone tested this to work? Jan ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] cannot replace X with Y: devices have different sector alignment
On Sat, Nov 10, 2012 at 8:14 AM, Trond Michelsen wrote: > When I try to replace the old drive, I get this error: > > # zpool replace tank c4t5000C5002AA2F8D6d0 c4t5000C5004DE863F2d0 > cannot replace c4t5000C5002AA2F8D6d0 with c4t5000C5004DE863F2d0: > devices have different sector alignment > > > How can I replace the drive without migrating all the data to a > different pool? It is possible, I hope? I had the same problem. I tried copying the partition layout and some other stuff but without success. I ended up having to recreate the pool and now have a non-mirrored root fs. If anyone has figured out how to mirror drives after getting the message about sector alignment, please let the list know :-). Jan ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] cannot replace X with Y: devices have different sector alignment
On Tue, Sep 25, 2012 at 6:42 PM, LIC mesh wrote: > The new drive I bought correctly identifies as 4096 byte blocksize! > So...OI doesn't like it merging with the existing pool. So... Any solution to this yet? I've got a 42 drive zpool (21 mirror vdevs) with 12 2TB drives that has 512byte blocksize. The remaining drives are 3TB with 4k blocksize, and the pool uses ashift=12. Recently this happened to one of the 2TB drives: mirror-13DEGRADED 0 0 0 c4t5000C5002AA2F8D6d0 UNAVAIL 0 0 0 cannot open c4t5000C5002AB4FF17d0 ONLINE 0 0 0 and even though it came back after a reboot, I'd like to swap it for a new drive. Obviously, all new drives have 4k blocksize, so I decided to replace both drives in the vdev with 3TB drives. The new drives are Seagate ST3000DM001-1CH1, and there are already 12 of these in the pool. # iostat -En ... c4t5000C5004DE1EFF2d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: ST3000DM001-1CH1 Revision: CC43 Serial No: Z1F0TKXV Size: 3000.59GB <3000592982016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 0 Predictive Failure Analysis: 0 c4t5000C5004DE863F2d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: ST3000DM001-1CH1 Revision: CC43 Serial No: Z1F0VHTG Size: 3000.59GB <3000592982016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 0 Predictive Failure Analysis: 0 c4t5000C5004DD3F76Bd0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: ST3000DM001-1CH1 Revision: CC43 Serial No: Z1F0T1QX Size: 3000.59GB <3000592982016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 0 Predictive Failure Analysis: 0 When I try to replace the old drive, I get this error: # zpool replace tank c4t5000C5002AA2F8D6d0 c4t5000C5004DE863F2d0 cannot replace c4t5000C5002AA2F8D6d0 with c4t5000C5004DE863F2d0: devices have different sector alignment How can I replace the drive without migrating all the data to a different pool? It is possible, I hope? -- Trond Michelsen ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
