SuperMicro has a number of servers built with infiniband on-baord in 1u, twin 1u and the like... not sureif they have onbaord infiniband with 4 pci-e 16x slots...
On Tue, Jun 30, 2009 at 1:14 AM, Michael Butash<mich...@butash.net> wrote: > Exactly as Nadeem said, start local, and if you need to expand, look at > infiniband. Buying IB gear directly from the vendor, supported, > warranted, etc, will get expensive quick. You'll probably find > programming info fairly obscure, as most of this is very proprietary, as > supercomputing is a cash cow to certain vars. > > Obscurity seems to make it cheap on the secondary market, as most app > dev's won't/don't take it to that level, and not many really need it. > Tesla's change things as they provide an extensible way to increase > generic crunching local, and can scale significantly with pcie bandwith > and hardware with enough slots. > > For me IB was interesting as I was investigating upping my house to > gige, fiber-channel, and other things to geek out on. IB actually > looked fairly promising as it's device stack allows for IP/FC > emululation layer, as well as it's own native socket stack where > parallel processing plays into. Then I found out it was cheaper than > buying decent 4g fiber channel gear off ebay, and got even more > interesting. Might provide some ideal scalability if you need to expand > the processing outside of one box eventually. > > -mb > > > On Tue, 2009-06-30 at 00:35 -0700, Stephen wrote: >> Are you looking at the 1u Tesla or the x58 personal supercomputer they >> are talking about... >> >> I would look into RH mostly for compatability fo ahrdware as it is >> tested by them, and for most packages... but the distro is really >> dependant on your app. >> >> Man they have really changed tesla, it used to be a bunch of gpu cards >> in a box chained to a header and then 16 of those can be tied together >> to a single machine. >> >> but now you can buy a preconfigured clsuter >> >> http://www.penguincomputing.com/products/linux/server/web_promotion/tesla_workgroup_cluster >> >> man i stop following for just 1 year... >> >> >> On Mon, Jun 29, 2009 at 6:29 PM, Eric Shubert<e...@shubes.net> wrote: >> > I'm looking at clustering together a handful of hosts, each running dual >> > nvidia tesla cards. Modeling applications of some sort. I honestly don't >> > know much more than that. >> > >> > Michael Butash wrote: >> >> You're probably talking infiniband switching, infiniband hba's, >> >> pci-e/htx interfaces, fiber channel disk arrays, etc. Linux seems to >> >> support infiniband hba's reasonably well, and 10g 4x infiniband hba's >> >> tend to be cheap these days on ebay. We're talking $100 used hba's for >> >> the nodes, and ~$1200 for a 12 port Cisco/Topspin switch. I thought >> >> about buying some to play with, as it ends up cheaper than IP or Fiber >> >> channel technologies, yet can replace them all to some extents. >> >> >> >> IB is quite versatile, emulating fiber-channel, IP network, or raw >> >> interrupt switching to a cpu and memory via different driver socket >> >> interface api's. Cray always used a similar means to make theirs with >> >> proprietary north bridge and software, but IB is more of a standard now, >> >> enabling (relatively) cheap supercomputing on the fly with commodity >> >> hardware. Well, hardware-wise at least... >> >> >> >> So yeah, what apps are you talking about utilizing it? >> >> >> >> -mb >> >> >> >> >> >> On Mon, 2009-06-29 at 13:47 -0700, Matt Graham wrote: >> >>> From: Eric Shubert <e...@shubes.net> >> >>>> Has anyone here implemented any clusters? >> >>> I've only set one up, but I maintain the ones that my predecessors >> >>> set up. It's not rocket science. >> >>> >> >>>> Is any particular distro better or worse at clustering? >> >>> Not really. Every distro has heartbeat/DRBD/LVS available. >> >>> >> >>>> Any pointers regarding clustering you'd like to share? >> >>> Define the problem you're trying to solve more rigorously than just >> >>> "clustering" first. Do you want flailover between 2 boxes? Do you >> >>> want a frontend box with N service-providing boxes behind it? The >> >>> answers to that greatly affect what you will end up doing, as does >> >>> the question "What services is this cluster going to provide?" >> >>> >> >>> Basically, all I can do is handwave without answers to "what services?" >> >>> and "how many machines?". >> >>> >> >> >> > >> > >> > -- >> > -Eric 'shubes' >> > >> > --------------------------------------------------- >> > PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us >> > To subscribe, unsubscribe, or to change your mail settings: >> > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss >> > >> >> >> > > --------------------------------------------------- > PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us > To subscribe, unsubscribe, or to change your mail settings: > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss > -- A mouse trap, placed on top of your alarm clock, will prevent you from rolling over and going back to sleep after you hit the snooze button. Stephen --------------------------------------------------- PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
