Hi, I have some strange goings-on with my VM of Solaris Express 11, and I hope someone can help.
It shares out other virtual machine files for use in ESXi 4.0 (it, too, runs in there) I had two disks inside the VM - one for rpool and one for 'vmpool'. All was fine. vmpool has some deduped data. That was also fine. I added a Samsung SSD to the ESXi host, created a 512MB VMDK and a 20GB VMDK, and added as log and cache, respectively. This also worked fine. At this point, the pool is made of c8t1d0 (data), c8t2d0 (logs), c8t3d0 (cache). I decide that to add some redundancy, I'll add a mirrored virtual disk. At this point, it happens that the VMDK for this disk (c8t4d0) actually resides on the same physical disk as c8t1d0. The idea was to perform the logical split in Solaris Express first, deal with the IO penalty of writing everything twice to the same physical disk (even though Solaris thinks they're two separate ones), then move that VMDK onto a separate physical disk shortly. This should in the short term protect against bit-flips and small errors on the single physical disk that ESXi has, until a second one is installed. I have a think about capacity, though, and decide I'd prefer the mirror to be of c8t4d0 and c8t5d0 instead. So, it seems I want to go from one single disk (c8t1d0), to a mirror of c8t4d0 and c8t5d0. In my mind, that's a 'zpool replace' onto c8t4d0 and a 'zpool attach' of c8t5d0. I kick off the replace, and all goes fine. Part way through I try to do the attach as well, but am politely told I can't. The replace itself completed without complaint, however on completion, virtual machines whose disks are inside 'vmpool' start hanging, checksum errors rapidly start counting up, and since there's no redundancy, nothing can be done to repair them. pool: vmpool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: resilvered 48.2G in 2h53m with 0 errors on Mon Jan 3 20:45:49 2011 config: NAME STATE READ WRITE CKSUM vmpool DEGRADED 0 0 25.6K c8t4d0 DEGRADED 0 0 25.6K too many errors logs c8t2d0 ONLINE 0 0 0 cache c8t3d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /vmpool/nfs/duck/duck_1-flat.vmdk /vmpool/nfs/panda/template.xppro-flat.vmdk At this point, I remove disk c8t1d0, and snapshot the entire VM in case I do any further damage. This leads to my first two questions: #1 - are there any suspicions as to what's happened here? How come the resilver completed fine but now there are checksum errors on the replacement disk? It does reside on the same physical disk, after all. Could this be something to do with me attempting the attach during the replace? #2 - in my mind, c8t1d0 contains the state of the pool just prior to the cutover to c8t4d0. Is there any way I can get this back, and scrap the contents of c8t4d0? A 'zpool import -D' is fruitless, but I imagine there's some way of tricking Solaris into seeing c8t1d0 this as a single disk pool again? Now that I've snapshotted the VM and have a sort of safety net, I run a scrub, which unsurprisingly unearths checksum errors and lists all of the files which have problems: pool: vmpool state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 0 in 0h30m with 95 errors on Mon Jan 3 21:47:25 2011 config: NAME STATE READ WRITE CKSUM vmpool ONLINE 0 0 190 c8t4d0 ONLINE 0 0 190 logs c8t2d0 ONLINE 0 0 0 cache c8t3d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /vmpool/nfs/duck/duck-flat.vmdk /vmpool/nfs/duck/Windows Server 2003 Standard Edition.nvram /vmpool/nfs/duck/duck_1-flat.vmdk /vmpool/nfs/eagle/eagle-flat.vmdk /vmpool/nfs/eagle/eagle_1-flat.vmdk /vmpool/nfs/eagle/eagle_2-flat.vmdk /vmpool/nfs/eagle/eagle_3-flat.vmdk /vmpool/nfs/eagle/eagle_5-flat.vmdk /vmpool/nfs/panda/Windows XP Professional.nvram /vmpool/nfs/panda/panda-flat.vmdk /vmpool/nfs/panda/template.xppro-flat.vmdk I 'zpool clear vmpool', power on one of the VMs, and the checksum count quickly reaches 970. #3 - why would this be the case? I thought the purpose of a scrub was to traverse all blocks, read them, and unearth problems? I'm wondering why these 970 errors haven't been found in the scrub? I power off the VM, perform another scrub. This time, 94 errors: pool: vmpool state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 0 in 0h33m with 94 errors on Mon Jan 3 22:27:30 2011 config: NAME STATE READ WRITE CKSUM vmpool ONLINE 0 0 1.13K c8t4d0 ONLINE 0 0 1.13K logs c8t2d0 ONLINE 0 0 0 cache c8t3d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /vmpool/nfs/duck/duck-flat.vmdk /vmpool/nfs/duck/duck_1-flat.vmdk /vmpool/nfs/eagle/eagle-flat.vmdk /vmpool/nfs/eagle/eagle_1-flat.vmdk /vmpool/nfs/eagle/eagle_2-flat.vmdk /vmpool/nfs/eagle/eagle_3-flat.vmdk /vmpool/nfs/eagle/eagle_5-flat.vmdk /vmpool/nfs/panda/Windows XP Professional.nvram /vmpool/nfs/panda/panda-flat.vmdk /vmpool/nfs/panda/template.xppro-flat.vmdk I then set the failmode of the pool to continue, in the hope that while ZFS thinks there are errors, the files would still be accessible over NFS and ESXi wouldn't care about them. Unsurprisingly, the VMs don't boot still. Can anyone help? I see the instructions on restoring files, but I'm quite surprised that a replace seems to have induced this problem. The last time I saw checksum problems was when I had a load of SATA disks behind USB controllers and I suffered power loss part way through a replace, so I can understand there's the potential to get into a mess. On this occasion there wasn't any power loss, and the event itself reported success .. ? Thank you in advance, Chris _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
