[OmniOS-discuss] Zpool export while resilvering?
List, This is probably a silly question, but I've honestly never tried this and don't have a test machine handy at the moment - can a pool be safely exported and re-imported later if it is currently resilvering? In the way of a bit of background, I have a pool made up with thirty or so 4TB Seagate disks with a firmware issue that results in their max temp being set at 40C as opposed to 60C. This particular pool is in an office building in Texas, in an air-conditioned server room. The condenser for this unit is in the building's plenum and when the building a/c goes off over weekends in the summer my server room a/c struggles and temps run up to about 85F or so. This is causing my pool to drop random disks lately (fmadm reports high temp and they get marked as removed from the pool), and I've only just narrowed it down to this firmware issue. Seagate firmware update utility is apparently Windows only, so the disks must come out for the firmware update, but the pool is resilvering several disks with days remaining, hence my original query. Regards, Rob ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] bug in powertop
Hi all, Anybody able to start powertop using some of the available options? root@nas:/root# powertop -d 1 Segmentation Fault (core dumped) root@nas:/root# powertop -t 10 Segmentation Fault (core dumped) root@nas:/root# powertop -d 1 -v Segmentation Fault (core dumped) root@nas:/root# powertop -c 0 Segmentation Fault (core dumped) The only way I am able to start powertop is using defaults. Omnios: OmniOS v11 r151014 -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael at rasmussen dot cc http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xD3C9A00E mir at datanom dot net http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE501F51C mir at miras dot org http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE3E80917 -- /usr/games/fortune -es says: He hated being thought of as one of those people that wore stupid ornamental armour. It was gilt by association. -- Terry Pratchett, Night Watch pgp68VbMudzj9.pgp Description: OpenPGP digital signature ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
This is probably a silly question, but I¹ve honestly never tried this and don¹t have a test machine handy at the moment can a pool be safely exported and re-imported later if it is currently resilvering? In the way of a bit of background, I have a pool made up with thirty or so 4TB Seagate disks with a firmware issue that results in their max temp being set at 40C as opposed to 60C. This particular pool is in an office building in Texas, in an air-conditioned server room. The condenser for this unit is in the building¹s plenum and when the building a/c goes off over weekends in the summer my server room a/c struggles and temps run up to about 85F or so. This is causing my pool to drop random disks lately (fmadm reports high temp and they get marked as removed from the pool), and I¹ve only just narrowed it down to this firmware issue. Seagate firmware update utility is apparently Windows only, so the disks must come out for the firmware update, but the pool is resilvering several disks with days remaining, hence my original query. Not an answer to your question, but the approach I'd take is renting a portable 110V or 220V A/C unit from somebody like spot-coolers.com to get you through the resilver, then apply the firmware update. (And then I'd start trying to convince management that it's worth adding a unit permanently -- our Office Pro 24 24000BTU/hr cost us under $4k back in 2011.) -- Dave Pooser Cat-Herder-in-Chief, Pooserville.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
You might also crank up the priority on your resilver, particularly if it is getting tripped all of the time: http://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/ -nld On Tue, Jun 9, 2015 at 1:31 PM, Dave Pooser dave...@pooserville.com wrote: This is probably a silly question, but I¹ve honestly never tried this and don¹t have a test machine handy at the moment can a pool be safely exported and re-imported later if it is currently resilvering? In the way of a bit of background, I have a pool made up with thirty or so 4TB Seagate disks with a firmware issue that results in their max temp being set at 40C as opposed to 60C. This particular pool is in an office building in Texas, in an air-conditioned server room. The condenser for this unit is in the building¹s plenum and when the building a/c goes off over weekends in the summer my server room a/c struggles and temps run up to about 85F or so. This is causing my pool to drop random disks lately (fmadm reports high temp and they get marked as removed from the pool), and I¹ve only just narrowed it down to this firmware issue. Seagate firmware update utility is apparently Windows only, so the disks must come out for the firmware update, but the pool is resilvering several disks with days remaining, hence my original query. Not an answer to your question, but the approach I'd take is renting a portable 110V or 220V A/C unit from somebody like spot-coolers.com to get you through the resilver, then apply the firmware update. (And then I'd start trying to convince management that it's worth adding a unit permanently -- our Office Pro 24 24000BTU/hr cost us under $4k back in 2011.) -- Dave Pooser Cat-Herder-in-Chief, Pooserville.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
On Jun 9, 2015, at 8:05 AM, Robert A. Brock robert.br...@2hoffshore.com wrote: List, This is probably a silly question, but I’ve honestly never tried this and don’t have a test machine handy at the moment – can a pool be safely exported and re-imported later if it is currently resilvering? yes. In the way of a bit of background, I have a pool made up with thirty or so 4TB Seagate disks with a firmware issue that results in their max temp being set at 40C as opposed to 60C. yep, this is the broken 003 firmware from Seagate, know it well :-P This particular pool is in an office building in Texas, in an air-conditioned server room. The condenser for this unit is in the building’s plenum and when the building a/c goes off over weekends in the summer my server room a/c struggles and temps run up to about 85F or so. This is causing my pool to drop random disks lately (fmadm reports high temp and they get marked as removed from the pool), and I’ve only just narrowed it down to this firmware issue. Seagate firmware update utility is apparently Windows only, so the disks must come out for the firmware update, but the pool is resilvering several disks with days remaining, hence my original query. fwflash might work, but it is unlikely Seagate knows anything about it. In any case, firmware upgrades on production system is not a best practice. You can also disable the FMA agent, disk-transport, which is the agent responsible for watching to ensure the temperature does not exceed the temperature at which the drive vendor says the drive should not be operated The impact to you is that the same agent detects other failures, such as predicted failures, that probably do need to be noticed. For a short window, this option might work for you. Useful commands: fmstat - shows the current FMA modules, and should include disk-transport fmadm unload disk-transport fmadm load disk-transport The temp checks (and PFA) are done once per hour, by default. -- richard Regards, Rob ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
On Jun 9, 2015, at 12:00 PM, Narayan Desai narayan.de...@gmail.com wrote: You might also crank up the priority on your resilver, particularly if it is getting tripped all of the time: http://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/ http://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/ -nld In general, yes this is a very good post. However, for more recent ZFS and certainly the lastest OmniOS something-14 release, the write throttle has been completely rewritten, positively impacting resilvers. And, with that rewrite, there is a few more tunables at your disposal, while the old ones fade to the bucket of bad memories :-) In most cases, resilver is capped by the time to write to the resilvering device. You can see this in iostat -x as the device that is 100% busy with write workload. That said, for this specific case, the drives are not actually failed, just taken offline, so you could have a short resilver session, once they are brought back online. -- richard On Tue, Jun 9, 2015 at 1:31 PM, Dave Pooser dave...@pooserville.com mailto:dave...@pooserville.com wrote: This is probably a silly question, but I¹ve honestly never tried this and don¹t have a test machine handy at the moment can a pool be safely exported and re-imported later if it is currently resilvering? In the way of a bit of background, I have a pool made up with thirty or so 4TB Seagate disks with a firmware issue that results in their max temp being set at 40C as opposed to 60C. This particular pool is in an office building in Texas, in an air-conditioned server room. The condenser for this unit is in the building¹s plenum and when the building a/c goes off over weekends in the summer my server room a/c struggles and temps run up to about 85F or so. This is causing my pool to drop random disks lately (fmadm reports high temp and they get marked as removed from the pool), and I¹ve only just narrowed it down to this firmware issue. Seagate firmware update utility is apparently Windows only, so the disks must come out for the firmware update, but the pool is resilvering several disks with days remaining, hence my original query. Not an answer to your question, but the approach I'd take is renting a portable 110V or 220V A/C unit from somebody like spot-coolers.com http://spot-coolers.com/ to get you through the resilver, then apply the firmware update. (And then I'd start trying to convince management that it's worth adding a unit permanently -- our Office Pro 24 24000BTU/hr cost us under $4k back in 2011.) -- Dave Pooser Cat-Herder-in-Chief, Pooserville.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com mailto:OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
They are failed as far as OmniOS is concerned, from what I can tell: Jun 08 01:08:54 710768e8-2f2b-4b3d-9d4b-a85ef5617219 DISK-8000-12 Major Host: 2hus291 Platform: S5500BC Chassis_id : Product_sn : Fault class : fault.io.disk.over-temperature Affects : dev:///:devid=id1,sd@n5000c5007242271f//scsi_vhci/disk@g5000c5007242271f faulted and taken out of service FRU : Slot 21 (hc://:product-id=LSI-SAS2X36:server-id=:chassis-id=500304800033213f:serial=S1Z02A8MK4361NF4:part=SEAGATE-ST4000NM0023:revision=0003/ses-enclosure=1/bay=20/disk=0) faulty Description : A disk's temperature exceeded the limits established by its manufacturer. Refer to http://illumos.org/msg/DISK-8000-12 for more information. root@2hus291:/root# zpool status pool0 pool: pool0 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Tue Jun 9 11:11:16 2015 18.8T scanned out of 91.7T at 667M/s, 31h48m to go 591G resilvered, 20.55% done config: NAME STATE READ WRITE CKSUM pool0DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 c0t5000C50055ECA49Bd0ONLINE 0 0 0 c0t5000C50055ECA4B3d0ONLINE 0 0 0 c0t5000C50055ECA587d0ONLINE 0 0 0 c0t5000C50055ECA6CFd0ONLINE 0 0 0 c0t5000C50055ECA7F3d0ONLINE 0 0 0 spare-5 REMOVED 0 0 0 c0t5000C5007242271Fd0 REMOVED 0 0 0 c0t5000C50055EF8A6Fd0 ONLINE 0 0 0 (resilvering) c0t5000C50055ECAB23d0ONLINE 0 0 0 c0t5000C50055ECABABd0ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 c0t5000C50055EE9D87d0ONLINE 0 0 0 c0t5000C50055EE9E43d0ONLINE 0 0 0 c0t5000C50055EEA5ABd0ONLINE 0 0 0 c0t5000C50055EEBA5Fd0ONLINE 0 0 0 c0t5000C50055EEC1E3d0ONLINE 0 0 0 c0t5000C500636670BFd0ONLINE 0 0 0 c0t5000C50055EF8CBBd0ONLINE 0 0 0 c0t5000C50055EF8D33d0ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 c0t5000C50055F7942Fd0ONLINE 0 0 0 c0t5000C50055F79E03d0ONLINE 0 0 0 c0t5000C50055F7A8DFd0ONLINE 0 0 0 c0t5000C50055F81C1Bd0ONLINE 0 0 0 c0t5000C5005604A42Bd0ONLINE 0 0 0 c0t5000C5005604A487d0ONLINE 0 0 0 c0t5000C5005604A74Bd0ONLINE 0 0 0 c0t5000C5005604A91Bd0ONLINE 0 0 0 raidz2-4 DEGRADED 0 0 0 c0t5000C500562ED6A3d0ONLINE 0 0 0 c0t5000C500562F8DEFd0ONLINE 0 0 0 c0t5000C500562F92D7d0ONLINE 0 0 0 c0t5000C500562FA0DFd0ONLINE 0 0 0 c0t5000C500636679EBd0ONLINE 0 0 0 spare-5 DEGRADED 0 014 c0t5000C50057FBB127d0 REMOVED 0 0 0 c0t5000C5006366906Bd0 ONLINE 0 0 0 c0t5000C5006366808Fd0ONLINE 0 0 0 spare-7 REMOVED 0 0 0 c0t5000C50057FC84F3d0 REMOVED 0 0 0 c0t5000C50063669937d0 ONLINE 0 0 0 logs mirror-3 ONLINE 0 0 0 c13t5003048000308398d0 ONLINE 0 0 0 c13t5003048000308399d0 ONLINE 0 0 0 cache c13t5E83A9705BC3d0 ONLINE 0 0 0 spares c0t5000C5006366906Bd0 INUSE currently in use c0t5000C50063669937d0 INUSE currently in use c0t5000C50055EF8A6Fd0 INUSE currently in use c0t5000C5006366994Bd0 AVAIL Seems to be this that’s got me: http://www.bigdatajunkie.com/index.php/10-hardware/19-seagate-constellation-es-3-firmware-0003 Can’t interact with them unless I bring them ‘out of retirement’: root@2hus291:/root# cat /etc/devices/retire_store
[OmniOS-discuss] kvm networking
This must have been done millions of times already, but as a newbie I'm having great difficulty getting a kvm guest inside a zone to network properly. It seems to me that I've tried every combination of configs I can think of (except one, no doubt ;) ) but can't get anything outside the zone to talk to the guest, with the exception of vnc and ssh (which I presume must be built into the kvm mechanism?). To the best of my knowledge (which isn't that much in this case), I've tried -net user, -net tap, -redir, hostfwd in addition to the -net nic as shown on the wiki. Linux help pages suggest it should be done by bridging, but create-bridge won't let me use the visible vnic as a link. After two days of head-scratching I've decided I need assistance; if anyone can please point me at a working methodology I'd appreciate it. Be warned though I might need it spelling out :) ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] kvm networking
In message 55775165.50...@thestephensdomain.com, Graham Stephens writes: This must have been done millions of times already, but as a newbie I'm having great difficulty getting a kvm guest inside a zone to network URL:http://lists.omniti.com/pipermail/omnios-discuss/2015-March/004477.html With 151014, you don't need to explicitly add /dev/kvm and /dev/dld John groenv...@acm.org ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Zpool export while resilvering?
I went through this problem a while back. There are some gotchas in getting them back online and firmware upgraded. The is will not talk to the drive until it has its firmware upgraded or cleared from the fault database. This drives will not flash with multipath enabled either. I ended up clearing the fault manager's database, disabling it and disconnecting half the SAS cables to get them flashed. -Chip On Jun 9, 2015 2:32 PM, Robert A. Brock robert.br...@2hoffshore.com wrote: They are failed as far as OmniOS is concerned, from what I can tell: Jun 08 01:08:54 710768e8-2f2b-4b3d-9d4b-a85ef5617219 DISK-8000-12 Major Host: 2hus291 Platform: S5500BC Chassis_id : Product_sn : Fault class : fault.io.disk.over-temperature Affects : dev:///:devid=id1,sd@n5000c5007242271f //scsi_vhci/disk@g5000c5007242271f faulted and taken out of service FRU : Slot 21 (hc://:product-id=LSI-SAS2X36:server-id=:chassis-id=500304800033213f:serial=S1Z02A8MK4361NF4:part=SEAGATE-ST4000NM0023:revision=0003/ses-enclosure=1/bay=20/disk=0) faulty Description : A disk's temperature exceeded the limits established by its manufacturer. Refer to http://illumos.org/msg/DISK-8000-12 for more information. root@2hus291:/root# zpool status pool0 pool: pool0 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Tue Jun 9 11:11:16 2015 18.8T scanned out of 91.7T at 667M/s, 31h48m to go 591G resilvered, 20.55% done config: NAME STATE READ WRITE CKSUM pool0DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 c0t5000C50055ECA49Bd0ONLINE 0 0 0 c0t5000C50055ECA4B3d0ONLINE 0 0 0 c0t5000C50055ECA587d0ONLINE 0 0 0 c0t5000C50055ECA6CFd0ONLINE 0 0 0 c0t5000C50055ECA7F3d0ONLINE 0 0 0 spare-5 REMOVED 0 0 0 c0t5000C5007242271Fd0 REMOVED 0 0 0 c0t5000C50055EF8A6Fd0 ONLINE 0 0 0 (resilvering) c0t5000C50055ECAB23d0ONLINE 0 0 0 c0t5000C50055ECABABd0ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 c0t5000C50055EE9D87d0ONLINE 0 0 0 c0t5000C50055EE9E43d0ONLINE 0 0 0 c0t5000C50055EEA5ABd0ONLINE 0 0 0 c0t5000C50055EEBA5Fd0ONLINE 0 0 0 c0t5000C50055EEC1E3d0ONLINE 0 0 0 c0t5000C500636670BFd0ONLINE 0 0 0 c0t5000C50055EF8CBBd0ONLINE 0 0 0 c0t5000C50055EF8D33d0ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 c0t5000C50055F7942Fd0ONLINE 0 0 0 c0t5000C50055F79E03d0ONLINE 0 0 0 c0t5000C50055F7A8DFd0ONLINE 0 0 0 c0t5000C50055F81C1Bd0ONLINE 0 0 0 c0t5000C5005604A42Bd0ONLINE 0 0 0 c0t5000C5005604A487d0ONLINE 0 0 0 c0t5000C5005604A74Bd0ONLINE 0 0 0 c0t5000C5005604A91Bd0ONLINE 0 0 0 raidz2-4 DEGRADED 0 0 0 c0t5000C500562ED6A3d0ONLINE 0 0 0 c0t5000C500562F8DEFd0ONLINE 0 0 0 c0t5000C500562F92D7d0ONLINE 0 0 0 c0t5000C500562FA0DFd0ONLINE 0 0 0 c0t5000C500636679EBd0ONLINE 0 0 0 spare-5 DEGRADED 0 014 c0t5000C50057FBB127d0 REMOVED 0 0 0 c0t5000C5006366906Bd0 ONLINE 0 0 0 c0t5000C5006366808Fd0ONLINE 0 0 0 spare-7 REMOVED 0 0 0 c0t5000C50057FC84F3d0 REMOVED 0 0 0 c0t5000C50063669937d0 ONLINE 0 0 0 logs mirror-3 ONLINE 0 0 0 c13t5003048000308398d0 ONLINE 0 0 0 c13t5003048000308399d0 ONLINE 0 0 0 cache c13t5E83A9705BC3d0 ONLINE 0 0 0
Re: [OmniOS-discuss] Zpool export while resilvering?
Fwflash has an issue: root@2hus291:/root# fwflash -f MegalodonES3-SAS-STD-0004.LOD -d /devices/pci@0,0/pci8086,340e@7/pci1000,3040@0/iport@f/disk@w5000c50057fbaf01,0:a,raw fwflash: sd-GENERIC firmware image verifier: supplied filename MegalodonES3-SAS-STD-0004.LOD exceeds maximum allowable size of 1468006 bytes Seems I’m not the first: https://www.illumos.org/issues/5012 Argh. I guess I have to pull the disks… From: Robert A. Brock Sent: 09 June 2015 20:31 To: omnios-discuss@lists.omniti.com Subject: RE: [OmniOS-discuss] Zpool export while resilvering? snip ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] kvm networking
Quoting Dominik Hassler hassl...@gmx.li: you need a dedicated vnic for the kvm. I assume you have one vnic and use that one for the zone so it is not exclusive for the kvm guest anymore and won't work to my knowledge. You can always bind VNC to a unix socket so you don't acutally need a vnic for the zone (except that one which is used by the zone but most not be configured within the zone at all). I found that if the vnic has an address for the zone, vnc works but the network access from the guest is sporadic at best. I tried with no address on the vnic, which means vnc doesn't work - but in my case that's not a problem as it's for an email server - and yes, outgoing connections work great, so it is probably how it should be. I can't talk to the email server from outside the zone, though. I don't know how I can pass the vnic to the zone without it being 'aware', unless you can add it as a device instead of a network interface? Did you have a look at https://github.com/hadfl/kvmadm ? It'll do all the necessary setup for you for running kvms within zones. I'll have to take a look at that, thanks. ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] bug in powertop
I gave this a go on 151012 (014 isn't available as an AMI yet, and I can't get the 006 AMI to accept a ssh key pair on instance creation), and saw the same thing. powertop appears to segfault in string_to_decimal / libc.so.1. I then tried rebuilding powertop from the OmniTI illumos-gate build scripts, but found that omnios-build:build/illumos/build.sh is a little fussy for the uninitiated. This sounds like a libc or kernel change, but I can't imagine why it would show up on both 012 and 014. Maybe someone with a working build environment can dig further? --Zach Malone On Tue, Jun 9, 2015 at 9:45 AM, Michael Rasmussen m...@miras.org wrote: Hi all, Anybody able to start powertop using some of the available options? root@nas:/root# powertop -d 1 Segmentation Fault (core dumped) root@nas:/root# powertop -t 10 Segmentation Fault (core dumped) root@nas:/root# powertop -d 1 -v Segmentation Fault (core dumped) root@nas:/root# powertop -c 0 Segmentation Fault (core dumped) The only way I am able to start powertop is using defaults. Omnios: OmniOS v11 r151014 -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael at rasmussen dot cc http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xD3C9A00E mir at datanom dot net http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE501F51C mir at miras dot org http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE3E80917 -- /usr/games/fortune -es says: He hated being thought of as one of those people that wore stupid ornamental armour. It was gilt by association. -- Terry Pratchett, Night Watch ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
