What network throughput were you looking at before the tweaking? Br, Rune
-----Original Message----- From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On Behalf Of Michael Rasmussen Sent: Sunday, November 09, 2014 5:21 PM To: omnios-discuss@lists.omniti.com Subject: Re: [OmniOS-discuss] infiniband On Mon, 10 Nov 2014 00:06:26 +0100 Michael Rasmussen <m...@miras.org> wrote: > > This explains the difference in speed since 2.5GT/s x2 is roughly 3.5 > gbps over IBoIP. > > Can anybody explain why Omnios only uses 2 lains when PCIe slot and > HCA is capable of 8 lains? > Found the "bug". All required was to change the slot;-) 01:00.0 InfiniBand: Mellanox Technologies MT25408 [ConnectX VPI - IB SDR / 10GigE] (rev a0) Subsystem: Mellanox Technologies Device 0003 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 11 Region 0: Memory at fea00000 (64-bit, non-prefetchable) Region 2: Memory at fc000000 (64-bit, prefetchable) Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] Vital Product Data Not readable Capabilities: [9c] MSI-X: Enable+ Count=256 Masked- Vector table: BAR=0 offset=0007c000 PBA: BAR=0 offset=0007d000 Capabilities: [60] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- And some network performance tuning: * network performance tuning set ndd:tcp_wscale_always=1 set ndd:tcp_tstamp_if_wscale=1 set ndd:tcp_max_buf=16777216 set ndd:tcp_cwnd_max=8388608 set ndd:tcp_xmit_hiwat=1048576 set ndd:tcp_recv_hiwat=1048576 /var/web-gui/data/tools/iperf/iperf -s -B 10.0.2.10 ------------------------------------------------------------ Server listening on TCP port 5001 Binding to local address 10.0.2.10 TCP window size: 1.00 MByte (default) ------------------------------------------------------------ [ 4] local 10.0.2.10 port 5001 connected with 10.0.2.1 port 35745 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 27.4 GBytes 7.84 Gbits/sec -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael <at> rasmussen <dot> cc http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E mir <at> datanom <dot> net http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C mir <at> miras <dot> org http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 -------------------------------------------------------------- /usr/games/fortune -es says: ROMEO: Courage, man; the hurt cannot be much. MERCUTIO: No, 'tis not so deep as a well, nor so wide as a church-door; but 'tis enough, 'twill serve. _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss