Robert,
Not directly answering your question, but just some general info:
Robert Dunkley wrote:
Config:
Tyan S4989 Quad Opteron
Opteron 8350
16Gb Ram
Mellanox 20Gb PCIE-X8 card
Direct Connect to identical system
Centos 5.2 & OFED 1.3.1
Results:
[EMAIL PROTECTED] device]# qperf 192.168.10.160 tcp_bw tcp_lat
tcp_bw:
bw = 671 MB/sec
Don't know what *exactly* is the max bw for tcp_bw (to me the
number looks ok), but note that the bottleneck here is IPoIB.
You have a quad-core CPU, so you can run four parallel tests,
and their combined bw would be similar to RDMA traffic bw,
which is 1.71 GB/s in your case.
tcp_lat:
latency = 31.1 us
[EMAIL PROTECTED] device]# qperf 192.168.10.160 rc_bi_bw
rc_bi_bw:
bw = 1.71 GB/sec
This one looks fine (I guess you have PCI Express Gen2, right?).
-- Yevgeny
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Robert
Dunkley
Sent: 24 July 2008 10:32
To: Or Gerlitz
Cc: [email protected]
Subject: RE: [ofa-general] Direct / "Crossover" IPOIB
Thanks Or,
I have installed and run opensm on one of the machines, I can now move
traffic between them.
Pings show a latency og 0.07-0.09ms which seems OK.
Qperf shows abysmal results in connected mode though:
[EMAIL PROTECTED] OFED-1.3.1]# qperf 192.168.10.161 tcp_bw tcp_lat
tcp_bw:
bw = 32.8 KB/sec
tcp_lat:
latency = 42.2 us
Datagram mode shows better bandwidth but still awful latency:
[EMAIL PROTECTED] OFED-1.3.1]# qperf 192.168.10.160 tcp_bw tcp_lat
tcp_bw:
bw = 400 MB/sec
tcp_lat:
latency = 41.2 us
Is there something wrong with my setup? (QPerf seems to think so)
Thanks,
Rob
-----Original Message-----
From: Or Gerlitz [mailto:[EMAIL PROTECTED]
Sent: 24 July 2008 09:44
To: Robert Dunkley
Cc: [email protected]
Subject: Re: [ofa-general] Direct / "Crossover" IPOIB
Robert Dunkley wrote:
I've just setup two machines linked by a single Inifiniband cable. The
IPOIB side of things looks OK, ifconfig shows the interface and each
machine can ping itself on its Infiniband IP. They cannot ping each
other though and ibstat looks wrong. What do I need to do to make this
setup work?
Port 1:
State: Initializing
Physical state: LinkUp
Base lid: 0
SM lid: 0
you have to run SM (Subnet Manager), if you issue
$ opensm
on one of the nodes, it will do that.
Or.
The SAQ Group
Registered Office: 18 Chapel Street, Petersfield, Hampshire. GU32 3DZ
SEMTEC Limited trading as SAQ is Registered in England & Wales
Company Number: 06481952
http://www.saqnet.co.uk AS29219
SAQ Group Delivers high quality, honestly priced communication and I.T.
services to UK Business.
DSL : Domains : Email : Hosting : CoLo : Servers : Racks : Transit :
Backups : Managed Networks : Remote Support.
Find us in http://www.thebestof.co.uk/petersfield
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
