James, On Wed, Jan 7, 2009 at 5:11 PM, James Lentini <[email protected]> wrote: > > I have a Mellanox ConnectX HCA connected to a Cisco SFS 7000D switch. > There are no other devices connected to the switch. > > According to their specifications, both the ConnectX chip (MT25418) in > my HCA and the InfiniScale III chip (MT47396) in my switch support a 4 > KB InfiniBand link MTU. > > In practice, I'm getting a 2 KB MTU. Is there some configuration > setting I need to change to enable a 4 KB MTU? I'd like to see how a 4 > KB MTU affects protocol performance. > > My topology looks like this: > > [HCA lid 8, port 1] <------> [SWITCH lid 2, port 1] > > smpquery shows a MtuCap of 4 KB for the HCA: > > # smpquery portinfo 8 1 > # Port info: Lid 8 port 1 > Mkey:............................0x0000000000000000 > GidPrefix:.......................0xfe80000000000000 > Lid:.............................0x0008 > SMLid:...........................0x0002 > CapMask:.........................0x2500868 > IsTrapSupported > IsAutomaticMigrationSupported > IsSLMappingSupported > IsSystemImageGUIDsupported > IsVendorClassSupported > IsCapabilityMaskNoticeSupported > IsClientRegistrationSupported > DiagCode:........................0x0000 > MkeyLeasePeriod:.................15 > LocalPort:.......................1 > LinkWidthEnabled:................1X or 4X > LinkWidthSupported:..............undefined (0) (IBA extension) > LinkWidthActive:.................4X > LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps > LinkState:.......................Active > PhysLinkState:...................LinkUp > LinkDownDefState:................Polling > ProtectBits:.....................0 > LMC:.............................0 > LinkSpeedActive:.................5.0 Gbps > LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps > NeighborMTU:.....................2048 > SMSL:............................0 > VLCap:...........................VL0-7 > InitType:........................0x00 > VLHighLimit:.....................4 > VLArbHighCap:....................8 > VLArbLowCap:.....................8 > InitReply:.......................0x00 > MtuCap:..........................4096 > VLStallCount:....................0 > HoqLife:.........................31 > OperVLs:.........................VL0-7 > PartEnforceInb:..................0 > PartEnforceOutb:.................0 > FilterRawInb:....................0 > FilterRawOutb:...................0 > MkeyViolations:..................0 > PkeyViolations:..................0 > QkeyViolations:..................0 > GuidCap:.........................32 > ClientReregister:................0 > SubnetTimeout:...................8 > RespTimeVal:.....................16 > LocalPhysErr:....................0 > OverrunErr:......................5 > MaxCreditHint:...................0 > RoundTrip:.......................0 > > but smpquery reports only a 2 KB MTU capability for the switch
If the switch ports support a 4K MTU cap, it needs to be advertised (via the PortInfo response). That's what needs fixing if possible rather than some SM policy. Without an MTUCap of 4K on both sides of the link, an SM cannot set NeighborMTU to 4K. -- Hal > # smpquery portinfo 2 1 > # Port info: Lid 2 port 1 > Mkey:............................0x0000000000000000 > GidPrefix:.......................0x0000000000000000 > Lid:.............................0x0000 > SMLid:...........................0x0000 > CapMask:.........................0x0 > DiagCode:........................0x0000 > MkeyLeasePeriod:.................0 > LocalPort:.......................1 > LinkWidthEnabled:................4X > LinkWidthSupported:..............1X or 4X > LinkWidthActive:.................4X > LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps > LinkState:.......................Active > PhysLinkState:...................LinkUp > LinkDownDefState:................Polling > ProtectBits:.....................0 > LMC:.............................0 > LinkSpeedActive:.................5.0 Gbps > LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps > NeighborMTU:.....................2048 > SMSL:............................0 > VLCap:...........................VL0-7 > InitType:........................0x00 > VLHighLimit:.....................0 > VLArbHighCap:....................8 > VLArbLowCap:.....................8 > InitReply:.......................0x00 > MtuCap:..........................2048 > VLStallCount:....................7 > HoqLife:.........................18 > OperVLs:.........................VL0-7 > PartEnforceInb:..................0 > PartEnforceOutb:.................0 > FilterRawInb:....................0 > FilterRawOutb:...................0 > MkeyViolations:..................0 > PkeyViolations:..................0 > QkeyViolations:..................0 > GuidCap:.........................0 > ClientReregister:................0 > SubnetTimeout:...................0 > RespTimeVal:.....................0 > LocalPhysErr:....................15 > OverrunErr:......................15 > MaxCreditHint:...................0 > RoundTrip:.......................0 > > and I confirmed that the switch had a MT47396: > > # smpquery nodeinfo 2 > # Node info: Lid 2 > BaseVers:........................1 > ClassVers:.......................1 > NodeType:........................Switch > NumPorts:........................24 > SystemGuid:......................0x0005ad03011df356 > Guid:............................0x0005ad00001df356 > PortGuid:........................0x0005ad00001df356 > PartCap:.........................8 > DevId:...........................0xb924 > Revision:........................0x000001a1 > LocalPort:.......................1 > VendorId:........................0x0005ad > _______________________________________________ > general mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
