Re: [OmniOS-discuss] infiniband
On Thu, Oct 23, 2014 at 10:36 PM, Michael Rasmussen m...@miras.org wrote: On Fri, 24 Oct 2014 12:08:14 +1100 David Bomba turbo...@gmail.com wrote: Correct, they are 40Gb/s cards. I have heard same stories of problems with 40Gb/s cards on Linux and FreeBSD. What it boils down to is that at this speed software subnet manager is causing problems so always use a switch with hardware subnet manager. This sounds apocryphal and wrong to me. I know of a several large infiniband networks (1k+ host ports, non-blocking) that run with subnet managers running on linux machines and generally work pretty well. A few remarks: - subnet managers are a part of the control plane, they don't participate directly in the data plane after initial connection setup. This means that the footprint of a QDR port is the same as an SDR port; data packets aren't processed through the subnet manager. - subnet managers aren't bug-free -- the versions integrated into switch firmware can't be upgraded a la carte, so you are dependent on the vendor to release a new version of the switch firmware to get bugfixes. - the processors included in switches are embedded and hence slow. this could easily make a difference to subnet performance, if you need to talk to the subnet manager frequently to get new path records. I'd strongly recommend against running switch provided subnet managers. -nld ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Ang: Re: infiniband
On 2014-10-24 08:37, Johan Kragsterman wrote: Hi, Michael! Interesting! Do you have any links to discussions or documentation of that? See this thread: http://forum.proxmox.com/threads/19789-Infiniband-40gbps-capped-at-12-5gbps This mail was virus scanned and spam checked before delivery. This mail is also DKIM signed. See header dkim-signature. ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Ang: Re: Ang: Re: infiniband
On Fri, 24 Oct 2014 13:17:39 +0200 Johan Kragsterman johan.kragster...@capvert.se wrote: Well, I can only find one person in that thread that BELIEVES that running SM on a switch is preferable to run OpenSM software. He hardly comes with any documentation of this, and I can't see any other that agrees... But of coarse, if you provide too little resources for OpenSM it will cause problems, or if you have any problems with the S/W or the configuration... I think the issue here is that OpenSM is a single threaded application so when the CPU which is assigned to it maxes out performance will drop. So I guess it comes down to a trade-off between the performance of the host CPU and the performance of the switch hardware. -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael at rasmussen dot cc http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xD3C9A00E mir at datanom dot net http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE501F51C mir at miras dot org http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE3E80917 -- /usr/games/fortune -es says: The default Magic Word, Abracadabra, actually is a corruption of the Hebrew phrase ha-Bracha dab'ra which means pronounce the blessing. pgp0_kr5bYyhZ.pgp Description: OpenPGP digital signature ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] switching from Stable to LTS
On Fri, 3 Oct 2014 01:11:31 +0200 Michael Rasmussen m...@miras.org wrote: I think the generel consensus is that unless you are very very very knowledgeable in the darkest corners of Illumos, having patience above and beyond the call of duty, have loads of spare time, and only have a server populated with data of absolut nonsense you should never even think of downgrading your server(s). OK but let's say despite all the warnings I were going to try anyway. Would it be possible to set up a LTS non-global zone on a machine on which Stable is installed? Michael. ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] best guide to tftp ?
On Oct 24, 2014, at 4:53 PM, Michael Mounteney gat...@landcroft.co.uk wrote: Hello, I want to use OmniOS as a boot server for various Linux thin clients. What's the most appropriate guide out there? There's a few, referring to obsolete problems, other OSs etc. This: http://omnios.omniti.com/wiki.php/KayakMultiRelease specifically talks about setting up for Kayak installation (PXE installation of OmniOS), but I suspect what's there can be modified to put entries into tftpboot/boot/grub/menu.lst that will help your Linux thin clients. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] switching from Stable to LTS
On Oct 24, 2014, at 4:56 PM, Michael Mounteney gat...@landcroft.co.uk wrote: On Fri, 3 Oct 2014 01:11:31 +0200 Michael Rasmussen m...@miras.org wrote: I think the generel consensus is that unless you are very very very knowledgeable in the darkest corners of Illumos, having patience above and beyond the call of duty, have loads of spare time, and only have a server populated with data of absolut nonsense you should never even think of downgrading your server(s). OK but let's say despite all the warnings I were going to try anyway. Would it be possible to set up a LTS non-global zone on a machine on which Stable is installed? No. The only way to even come close to approaching that would be to run a KVM zone, and put OmniOS LTS in that. There are also pitfalls in doing so (similar to the pitfalls of running TCP over TCP... in this case, running ZFS volumes that will become proper ZFS filesystems). You do realize that the next stable OmniOS is also going to be the next LTS release, right? If you can manage until springtime, there will be a new LTS release, and I suspect much of omniti-ms will be appropriately updated. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS Logical Unit stuck in the offlining state with COMSTAR
On Oct 24, 2014, at 5:22 PM, sergei serge...@gmail.com wrote: I'm having same issue with OmniOS v11 r151006. Found few threads on this including http://lists.omniti.com/pipermail/omnios-discuss/2013-March/000537.html and https://illumos.org/issues/3621 but still can't figure out whats going on. Basic symptoms are that some zvol-backed LUs can not be deleted. And stmfadm list-lu -v LU reports LU state as Offlining. For what its worth we run same Dell R720xd as one of the earlier posters. I would really appreciate few tips on how to resolve this or if COMSTAR / stmf is more reliable in some other Illumos flavor ? You can try r151012, which has many bugfixes in it, some of which are COMSTAR/stmf related. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Ang: Re: Ang: Re: infiniband
ConnectX-2 cards work very well with OmniOS, at least over SRP, and those HCA's can be found for next to nothing. Comstar rules. I feel there's some new momentum behind Infiniband, and hoping that a new driver for the latest hardware will end up in Illumos sometime in the near future (I'm willing to pay a fair bounty). That said, QDR is still extremely fast and can easily co-mingle with FDR fabrics. The subnet manager can be any linux or windows box sitting on the fabric, no need for a managed switch in my experience, although my fabrics are modestly sized and quite stable. To reiterate what others have said, the subnet manager has nothing to do with target/initiator data transfer. OpenSM just scans the fabric at regular intervals for topology changes and handles session setup and teardown. I think of OpenSM as a sort of DHCP server for Infiniband, and in that analogy, it would be strange to believe that even the most resource-deprived DHCP server would affect data transfer on a network. A managed switch is akin to a DHCP server embedded in a router. Convenient, but not necessary. Greg, I'll probably contact you off-list to learn about your FreeBSD implementation. My ConnectX-3 FDR fabric is currently ZoL-based because I couldn't find concrete documentation on the ULP's available in FreeBSD OFED, whereas Linux is quite straightforward, albeit arguably not the most robust ZFS implementation. I'm doing Linux zvol's exported over SRP via SCST on CentOS to a mix of Linux and Windows initiators and it's been rock-solid for about a year now. ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS Logical Unit stuck in the offlining state with COMSTAR
Does r151012 include Illumos 3464 commit ? I went to the latest r151013 and that proved to be a very radical change. We've spent a lot to tune performance and make this system work well. And with 3464 all TXG logic has changed. Sergei On Fri, Oct 24, 2014 at 2:26 PM, Dan McDonald dan...@omniti.com wrote: On Oct 24, 2014, at 5:22 PM, sergei serge...@gmail.com wrote: I'm having same issue with OmniOS v11 r151006. Found few threads on this including http://lists.omniti.com/pipermail/omnios-discuss/2013-March/000537.html and https://illumos.org/issues/3621 but still can't figure out whats going on. Basic symptoms are that some zvol-backed LUs can not be deleted. And stmfadm list-lu -v LU reports LU state as Offlining. For what its worth we run same Dell R720xd as one of the earlier posters. I would really appreciate few tips on how to resolve this or if COMSTAR / stmf is more reliable in some other Illumos flavor ? You can try r151012, which has many bugfixes in it, some of which are COMSTAR/stmf related. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
