Re: [openib-general] Multi-port HCA

2006-10-06 Thread Shannon V. Davidson
john t wrote: Hi Shannon,   The bandwidth figures that you quoted below match with my readings for single port Mellanox DDR HCA (both for unidirection and bidirection). So it seems dual port SDR HCA performs as good as single port DDR HCA. It would help if you can also tell the bandwidth

Re: [openib-general] Multi-port HCA

2006-10-06 Thread Michael Krause
Off-line someone asked me to clarify my earlier e-mail.  Given this discussion continues, perhaps this might help explain the performance a bit more.  The Max Payload Size quoted here is what is typically implemented on x86 chipsets though other chipsets may use a larger value.  From a pure band

Re: [openib-general] Multi-port HCA

2006-10-06 Thread john t
Hi Shannon,   The bandwidth figures that you quoted below match with my readings for single port Mellanox DDR HCA (both for unidirection and bidirection). So it seems dual port SDR HCA performs as good as single port DDR HCA. It would help if you can also tell the bandwidth that you got using one p

Re: [openib-general] Multi-port HCA

2006-10-05 Thread Michael Krause
At 07:18 AM 10/5/2006, Roland Dreier wrote: > Bernard> I don't think it is the PCI-e bus because it can handle > Bernard> much more than 20 Gb/s. > >This isn't true. Mellanox cards have PCI-e x8 interfaces, which has a >theoretical limit of 16 Gb/sec in each direction, and a practical >lim

Re: [openib-general] Multi-port HCA

2006-10-05 Thread Shannon V. Davidson
John, In our testing with dual port Mellanox SDR HCAs, we found that not all PCI-express implementations are equal.  Depending on the PCIe chipset, we measured unidirectional SDR dual-rail bandwidth ranging from 1100-1500 MB/sec and bidirectional SDR dual-rail bandwidth ranging from 1570-2600

Re: [openib-general] Multi-port HCA

2006-10-05 Thread Bernard King-Smith
Roland Dreier <[EMAIL PROTECTED]> wrote on 10/05/2006 10:18:49 AM: >     Bernard> I don't think it is the PCI-e bus because it can handle >     Bernard> much more than 20 Gb/s. > > This isn't true.  Mellanox cards have PCI-e x8 interfaces, which has a > theoretical limit of 16 Gb/sec in each dir

Re: [openib-general] Multi-port HCA

2006-10-05 Thread Roland Dreier
Bernard> I don't think it is the PCI-e bus because it can handle Bernard> much more than 20 Gb/s. This isn't true. Mellanox cards have PCI-e x8 interfaces, which has a theoretical limit of 16 Gb/sec in each direction, and a practical limit that is even lower due to packetization and other

Re: [openib-general] Multi-port HCA

2006-10-05 Thread Bernard King-Smith
"john t" <[EMAIL PROTECTED]> wrote on 10/05/2006 08:18:31 AM: > Hi Bernard, >   > I had a configuration issue. I fixed it and now I get same BW (i.e. > around 10 Gb/sec) on each port provided I use ports on different HCA > cards. If I use two ports of the same HCA card then BW gets divided > be

Re: [openib-general] Multi-port HCA

2006-10-05 Thread john t
Hi Bernard,   I had a configuration issue. I fixed it and now I get same BW (i.e. around 10 Gb/sec) on each port provided I use ports on different HCA cards. If I use two ports of the same HCA card then BW gets divided between these two ports. I am using Mellanox HCA cards and doing simple send/rec

Re: [openib-general] Multi-port HCA

2006-10-03 Thread Bernard King-Smith
John, Who's adapter (manufacturer) are you using? It is usually an adapter implementation or driver issue that occures when you cannot scale across multiple links. The fact that you don't scale up from one link, but it appears they share a fixed bandwidth across N links means that there is a driv

[openib-general] Multi-port HCA

2006-10-03 Thread john t
Hi,   I have two HCA cards, each having two ports and each connected to a separate PCI-E x8 slot.   Using one HCA port I get end to end BW of 11.6 Gb/sec (uni-direction RDMA). If I use two ports of the same HCA or different HCA, I get between 5 to 6.5 Gb/sec point-to-point BW on each port. BW on ea