Re: [OmniOS-discuss] Zpool export while resilvering?
On Wed, Jun 10, 2015 at 4:09 AM, Robert A. Brock robert.br...@2hoffshore.com wrote: What did you use to flash them? Fwflash just gives an error about firmware file being too large. Santools. -Chip *From:* Schweiss, Chip [mailto:c...@innovates.com] *Sent:* 09 June 2015 21:25 *To:* Robert A. Brock *Cc:* omnios-discuss *Subject:* Re: [OmniOS-discuss] Zpool export while resilvering? I went through this problem a while back. There are some gotchas in getting them back online and firmware upgraded. The is will not talk to the drive until it has its firmware upgraded or cleared from the fault database. This drives will not flash with multipath enabled either. I ended up clearing the fault manager's database, disabling it and disconnecting half the SAS cables to get them flashed. -Chip snip 2H Offshore Engineering Ltd | Registered in England No. 02790139 | Registered office: Ferryside, Ferry Road, Norwich NR1 1SW. 2H Offshore is an Acteon company specializing in the design, monitoring and integrity management of offshore riser and conductor systems. Acteon is a group of specialist international engineering companies serving the offshore oil and gas industry. Its focus is on subsea services spanning the entire life of field. For more information, visit www.acteon.com The information in and/or accompanying this email is intended for the use of the stated recipient only and may be confidential and/or privileged. It should not be forwarded or copied nor should its contents be disclosed in any manner without the express consent of the sender/author. Any views or opinions presented are solely those of the author and do not necessarily represent those of 2H. ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
What did you use to flash them? Fwflash just gives an error about firmware file being too large. From: Schweiss, Chip [mailto:c...@innovates.com] Sent: 09 June 2015 21:25 To: Robert A. Brock Cc: omnios-discuss Subject: Re: [OmniOS-discuss] Zpool export while resilvering? I went through this problem a while back. There are some gotchas in getting them back online and firmware upgraded. The is will not talk to the drive until it has its firmware upgraded or cleared from the fault database. This drives will not flash with multipath enabled either. I ended up clearing the fault manager's database, disabling it and disconnecting half the SAS cables to get them flashed. -Chip snip ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
This is probably a silly question, but I¹ve honestly never tried this and don¹t have a test machine handy at the moment can a pool be safely exported and re-imported later if it is currently resilvering? In the way of a bit of background, I have a pool made up with thirty or so 4TB Seagate disks with a firmware issue that results in their max temp being set at 40C as opposed to 60C. This particular pool is in an office building in Texas, in an air-conditioned server room. The condenser for this unit is in the building¹s plenum and when the building a/c goes off over weekends in the summer my server room a/c struggles and temps run up to about 85F or so. This is causing my pool to drop random disks lately (fmadm reports high temp and they get marked as removed from the pool), and I¹ve only just narrowed it down to this firmware issue. Seagate firmware update utility is apparently Windows only, so the disks must come out for the firmware update, but the pool is resilvering several disks with days remaining, hence my original query. Not an answer to your question, but the approach I'd take is renting a portable 110V or 220V A/C unit from somebody like spot-coolers.com to get you through the resilver, then apply the firmware update. (And then I'd start trying to convince management that it's worth adding a unit permanently -- our Office Pro 24 24000BTU/hr cost us under $4k back in 2011.) -- Dave Pooser Cat-Herder-in-Chief, Pooserville.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
You might also crank up the priority on your resilver, particularly if it is getting tripped all of the time: http://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/ -nld On Tue, Jun 9, 2015 at 1:31 PM, Dave Pooser dave...@pooserville.com wrote: This is probably a silly question, but I¹ve honestly never tried this and don¹t have a test machine handy at the moment can a pool be safely exported and re-imported later if it is currently resilvering? In the way of a bit of background, I have a pool made up with thirty or so 4TB Seagate disks with a firmware issue that results in their max temp being set at 40C as opposed to 60C. This particular pool is in an office building in Texas, in an air-conditioned server room. The condenser for this unit is in the building¹s plenum and when the building a/c goes off over weekends in the summer my server room a/c struggles and temps run up to about 85F or so. This is causing my pool to drop random disks lately (fmadm reports high temp and they get marked as removed from the pool), and I¹ve only just narrowed it down to this firmware issue. Seagate firmware update utility is apparently Windows only, so the disks must come out for the firmware update, but the pool is resilvering several disks with days remaining, hence my original query. Not an answer to your question, but the approach I'd take is renting a portable 110V or 220V A/C unit from somebody like spot-coolers.com to get you through the resilver, then apply the firmware update. (And then I'd start trying to convince management that it's worth adding a unit permanently -- our Office Pro 24 24000BTU/hr cost us under $4k back in 2011.) -- Dave Pooser Cat-Herder-in-Chief, Pooserville.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
On Jun 9, 2015, at 8:05 AM, Robert A. Brock robert.br...@2hoffshore.com wrote: List, This is probably a silly question, but I’ve honestly never tried this and don’t have a test machine handy at the moment – can a pool be safely exported and re-imported later if it is currently resilvering? yes. In the way of a bit of background, I have a pool made up with thirty or so 4TB Seagate disks with a firmware issue that results in their max temp being set at 40C as opposed to 60C. yep, this is the broken 003 firmware from Seagate, know it well :-P This particular pool is in an office building in Texas, in an air-conditioned server room. The condenser for this unit is in the building’s plenum and when the building a/c goes off over weekends in the summer my server room a/c struggles and temps run up to about 85F or so. This is causing my pool to drop random disks lately (fmadm reports high temp and they get marked as removed from the pool), and I’ve only just narrowed it down to this firmware issue. Seagate firmware update utility is apparently Windows only, so the disks must come out for the firmware update, but the pool is resilvering several disks with days remaining, hence my original query. fwflash might work, but it is unlikely Seagate knows anything about it. In any case, firmware upgrades on production system is not a best practice. You can also disable the FMA agent, disk-transport, which is the agent responsible for watching to ensure the temperature does not exceed the temperature at which the drive vendor says the drive should not be operated The impact to you is that the same agent detects other failures, such as predicted failures, that probably do need to be noticed. For a short window, this option might work for you. Useful commands: fmstat - shows the current FMA modules, and should include disk-transport fmadm unload disk-transport fmadm load disk-transport The temp checks (and PFA) are done once per hour, by default. -- richard Regards, Rob ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
On Jun 9, 2015, at 12:00 PM, Narayan Desai narayan.de...@gmail.com wrote: You might also crank up the priority on your resilver, particularly if it is getting tripped all of the time: http://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/ http://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/ -nld In general, yes this is a very good post. However, for more recent ZFS and certainly the lastest OmniOS something-14 release, the write throttle has been completely rewritten, positively impacting resilvers. And, with that rewrite, there is a few more tunables at your disposal, while the old ones fade to the bucket of bad memories :-) In most cases, resilver is capped by the time to write to the resilvering device. You can see this in iostat -x as the device that is 100% busy with write workload. That said, for this specific case, the drives are not actually failed, just taken offline, so you could have a short resilver session, once they are brought back online. -- richard On Tue, Jun 9, 2015 at 1:31 PM, Dave Pooser dave...@pooserville.com mailto:dave...@pooserville.com wrote: This is probably a silly question, but I¹ve honestly never tried this and don¹t have a test machine handy at the moment can a pool be safely exported and re-imported later if it is currently resilvering? In the way of a bit of background, I have a pool made up with thirty or so 4TB Seagate disks with a firmware issue that results in their max temp being set at 40C as opposed to 60C. This particular pool is in an office building in Texas, in an air-conditioned server room. The condenser for this unit is in the building¹s plenum and when the building a/c goes off over weekends in the summer my server room a/c struggles and temps run up to about 85F or so. This is causing my pool to drop random disks lately (fmadm reports high temp and they get marked as removed from the pool), and I¹ve only just narrowed it down to this firmware issue. Seagate firmware update utility is apparently Windows only, so the disks must come out for the firmware update, but the pool is resilvering several disks with days remaining, hence my original query. Not an answer to your question, but the approach I'd take is renting a portable 110V or 220V A/C unit from somebody like spot-coolers.com http://spot-coolers.com/ to get you through the resilver, then apply the firmware update. (And then I'd start trying to convince management that it's worth adding a unit permanently -- our Office Pro 24 24000BTU/hr cost us under $4k back in 2011.) -- Dave Pooser Cat-Herder-in-Chief, Pooserville.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com mailto:OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
-store-magic▒▒`(rio-store-flagsP/scsi_vhci/disk@g5000c50057fbaf030rio-store-version(rio-store-magic▒▒`(rio-store-flagsP/scsi_vhci/disk@g5000c50057fbb1270rio-store-version(rio-store-magic▒▒`(rio-store-flagsP/scsi_vhci/disk@g5000c50057fc84f30rio-store-version(rio-store-magic▒▒`(rio-store-flagsP/scsi_vhci/disk@g5000c5007242271f0rio-store-version(rio-store-magic▒▒`(rio-store-flags Which, as I understand it, will necessitate a reboot. I’m going to try fwflash to see if I can get the firmware upgraded without having to pull all the disks and hook them up to a windows machine – has anyone else tried fwflash and Seagate disk firmware with any success? Fwflash seems to be able to see the disks, so I’m hopeful: root@2hus291:/root# fwflash -l | grep ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On Behalf Of Richard Elling Sent: 09 June 2015 20:17 To: Narayan Desai Cc: omnios-discuss@lists.omniti.com Subject: Re: [OmniOS-discuss] Zpool export while resilvering? On Jun 9, 2015, at 12:00 PM, Narayan Desai narayan.de...@gmail.commailto:narayan.de...@gmail.com wrote: You might also crank up the priority on your resilver, particularly if it is getting tripped all of the time: http://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/ -nld In general, yes this is a very good post. However, for more recent ZFS and certainly the lastest OmniOS something-14 release, the write throttle has been completely rewritten, positively impacting resilvers. And, with that rewrite, there is a few more tunables at your disposal, while the old ones fade to the bucket of bad memories :-) In most cases, resilver is capped by the time to write to the resilvering device. You can see this in iostat -x as the device that is 100% busy with write workload. That said, for this specific case, the drives are not actually failed, just taken offline, so you could have a short resilver session, once they are brought back online. -- richard This is probably a silly question, but I¹ve honestly never tried this and don¹t have a test machine handy at the moment can a pool be safely exported and re-imported later if it is currently resilvering? ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
spares c0t5000C5006366906Bd0 INUSE currently in use c0t5000C50063669937d0 INUSE currently in use c0t5000C50055EF8A6Fd0 INUSE currently in use c0t5000C5006366994Bd0 AVAIL Seems to be this that’s got me: http://www.bigdatajunkie.com/index.php/10-hardware/19-seagate-constellation-es-3-firmware-0003 Can’t interact with them unless I bring them ‘out of retirement’: root@2hus291:/root# cat /etc/devices/retire_store ▒ܱY^P/scsi_vhci/disk@g5000c500724eacb70rio-store-version(rio-store-magic ▒▒`(rio-store-flagsP/scsi_vhci/disk@g5000c50057fbc1c30rio-store-version (rio-store-magic▒▒ `(rio-store-flagsP/scsi_vhci/disk@g5000c50057fbaf030rio-store-version (rio-store-magic▒▒ `(rio-store-flagsP/scsi_vhci/disk@g5000c50057fbb1270rio-store-version (rio-store-magic▒▒ `(rio-store-flagsP/scsi_vhci/disk@g5000c50057fc84f30rio-store-version (rio-store-magic▒▒ `(rio-store-flagsP/scsi_vhci/disk@g5000c5007242271f0rio-store-version (rio-store-magic▒▒`(rio-store-flags Which, as I understand it, will necessitate a reboot. I’m going to try fwflash to see if I can get the firmware upgraded without having to pull all the disks and hook them up to a windows machine – has anyone else tried fwflash and Seagate disk firmware with any success? Fwflash seems to be able to see the disks, so I’m hopeful: root@2hus291:/root# fwflash -l | grep ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 Product : ST4000NM0023 *From:* OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] *On Behalf Of *Richard Elling *Sent:* 09 June 2015 20:17 *To:* Narayan Desai *Cc:* omnios-discuss@lists.omniti.com *Subject:* Re: [OmniOS-discuss] Zpool export while resilvering? On Jun 9, 2015, at 12:00 PM, Narayan Desai narayan.de...@gmail.com wrote: You might also crank up the priority on your resilver, particularly if it is getting tripped all of the time: http://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/ -nld In general, yes this is a very good post. However, for more recent ZFS and certainly the lastest OmniOS something-14 release, the write throttle has been completely rewritten, positively impacting resilvers. And, with that rewrite, there is a few more tunables at your disposal, while the old ones fade to the bucket of bad memories :-) In most cases, resilver is capped by the time to write to the resilvering device. You can see this in iostat -x as the device that is 100% busy with write workload. That said, for this specific case, the drives are not actually failed, just taken offline, so you could have a short resilver session, once they are brought back online. -- richard This is probably a silly question, but I¹ve honestly never tried this and don¹t have a test machine handy at the moment can a pool be safely exported and re-imported later if it is currently resilvering? ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
Fwflash has an issue: root@2hus291:/root# fwflash -f MegalodonES3-SAS-STD-0004.LOD -d /devices/pci@0,0/pci8086,340e@7/pci1000,3040@0/iport@f/disk@w5000c50057fbaf01,0:a,raw fwflash: sd-GENERIC firmware image verifier: supplied filename MegalodonES3-SAS-STD-0004.LOD exceeds maximum allowable size of 1468006 bytes Seems I’m not the first: https://www.illumos.org/issues/5012 Argh. I guess I have to pull the disks… From: Robert A. Brock Sent: 09 June 2015 20:31 To: omnios-discuss@lists.omniti.com Subject: RE: [OmniOS-discuss] Zpool export while resilvering? snip ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
