hi Greg, Very useful info! I already was wondering about the different timings i see for infiniband, but indeed it's the ConnectX that scores better in latency.
$289 on ebay but that's directly QDR then. "ConnectX-2 Dual-Port VPI QDR Infiniband Mezzanine I/O Card for Dell PowerEdge M1000e-Series Blade Servers" This 1.91 microseconds for a RDMA read is for a connectx. Not bad for Infiniband. Only 50% slower in latency than quadrics which is pci-x of course. Yet now needed is a cheap price for 'em :) It seems indeed all the 'cheap' offers are the infinihost III DDR versions. Regards, Vincent On Nov 7, 2011, at 9:21 PM, Greg Keller wrote: > >> Date: Mon, 07 Nov 2011 13:16:00 -0500 >> From: Prentice Bisbal<[email protected]> >> Subject: Re: [Beowulf] building Infiniband 4x cluster questions >> Cc: Beowulf Mailing List<[email protected]> >> Message-ID:<[email protected]> >> Content-Type: text/plain; charset=ISO-8859-1 >> >> Vincent, >> >> Don't forget that between SDR and QDR, there is DDR. If SDR is too >> slow, and QDR is too expensive, DDR might be just right. > And for DDR a key thing is, when latency matters, "ConnectX" DDR is > much > better than the earlier "Infinihost III" DDR cards. We have 100's of > each and the ConnectX make a large impact for some codes. Although > nearly antique now, we actually have plans for the ConnectX cards > in yet > another round of updated systems. This is the 3rd Generation system I > have been able to re-use the cards in (Harperton, Nehalem, and now > Single Socket Sandy Bridge), which makes me very happy. A great > investment that will likely live until PCI-Gen3 slots are the norm. > -- > Da Bears?! > >> -- >> Goldilocks >> >> >> On 11/07/2011 11:58 AM, Vincent Diepeveen wrote: >>>> hi Prentice, >>>> >>>> I had noticed the diff between SDR up to QDR, >>>> the SDR cards are affordable, the QDR isn't. >>>> >>>> The SDR's are all $50-$75 on ebay now. The QDR's i didn't find >>>> cheap >>>> prices in that pricerange yet. >>>> >>>> If i would want to build a network that's low latency and had a >>>> budget >>>> of $800 or so a node of course i would >>>> build a dolphin SCI network, as that's probably the fastest >>>> latency >>>> card sold for a $675 or so a piece. >>>> >>>> I do not really see a rival latency wise to Dolphin there. I >>>> bet most >>>> manufacturers selling clusters don't use >>>> it as they can make $100 more profit or so selling other >>>> networking >>>> stuff, and universities usually swallow that. >>>> >>>> So price total dominates the network. As it seems now >>>> infiniband 4x is >>>> not going to offer enough performance. >>>> The one-way pingpong latencies over a switch that i see of it, >>>> are not >>>> very convincing. I see remote writes to RAM >>>> are like nearly 10 microseconds for 4x infiniband and that card >>>> is the >>>> only one affordable. >>>> >>>> The old QM400's i have here are one-way pingpong 2.1 us or so, and >>>> QM500-B's are plentyful on the net (of course big disadvantage: >>>> needs >>>> pci-x), >>>> which are a 1.3 us or so there and have SHMEM. Not seeing a cheap >>>> switch for the QM500's though nor cables. >>>> >>>> You see price really dominates everything here. Small cheap >>>> nodes you >>>> cannot build if the port price, thanks to expensive network card, >>>> more than doubles. >>>> >>>> Power is not the real concern for now - if a factory already >>>> burns a >>>> couple of hundreds of megawatts, a small cluster somewhere on the >>>> attick eating >>>> a few kilowatts is not really a problem:) >>>> > > _______________________________________________ > Beowulf mailing list, [email protected] sponsored by Penguin > Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
