Re: [openib-general] basic IB doubt
Date: Mon, 28 Aug 2006 10:22:52 -0600 From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] CC: [EMAIL PROTECTED]; openib-general@openib.org Subject: Re: [openib-general] basic IB doubt OnMon,Aug28,2006at10:38:43AM-0400,Talpey,Thomaswrote: WillturningontheOpteron'sIOMMUintroducesomeofthese issuestox86? No,definatelynot.TheOpteronIOMMU(theGART)isapureaddress translationmechanismanddoesn'tchangetheoperationofthecaches. IfSunhasaproblemonlargersystemsIwonderifSGIAltixalsohasa problem?SGIAltixisdefinatelyarealsystemthatpeopleuseIB cardsintodayanditwouldbeeasytoimaginesuchalargesystem couldhavecoherenceissueswithmemorypolling.. Jason Yes, there's an issue with polling on the last byte of data, but not polling ona properlyimplementedCQ. The SGI machines have fully cache coherent I/O,e.g. as part of a DMA writean I/O controller will invalidate any cached copies ofa cache line being DMA written. There is still an ordering issue with respect to polling on the last byte of data. For example assume an incoming RDMA WRITE results in two DMA writes to two different pages, then the ordering of these DMA writevis-a-vis being visible in the coherent domain is not guaranteed. Instead these machines have an I/O sync operation to implement ordering guarantees,e.g.when such an I/O syncis used in our example to implement the CQthen it's guaranteed that both DMA writes have completed vis-a-vis being visible in the coherent domain. Hope that helps, Asgeir Eiriksson CTO Chelsio Communications, Inc.Express yourself with gadgets on Windows Live Spaces Try it! ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: QoS RFC
Roland This thread also indirectly brings up the issue of OpenFabrics, RNIC, and QoS. The RNIC devices don't have to be, but are typically unified wire devices, i.e. have simultaneous support for regular Ethernet NIC functions such as LSO/TSO and checksum offloads; iSCSI HBA initiator and/or target functionality; RNIC functionality; and TCP/IP offload. All the usual Ethernet TCP/IP management and configuration tools work as expected for such a device, e.g. for the purpose of this discussion QoS configuration with DiffServ, etc. At the same time: if there's an OpenFabrics QoS API, then this API needs to be transport agnostic, such that RNIC QoS can be configured through this same OpenFabrics QoS API. To be clear: the QoS requirements presented in the RFC look quite reasonable for an RNIC device, but the initial proposal didn't abstract out non-essential (to QoS) IB detail. Regards, Asgeir Eiriksson CTO Chelsio Communications Inc. -Original Message- From: [EMAIL PROTECTED] [mailto:openib-general- [EMAIL PROTECTED] On Behalf Of Roland Dreier Sent: Tuesday, May 30, 2006 7:44 AM To: Eitan Zahavi Cc: Nimrod Gindi; Roland Dreier; openib-general@openib.org Subject: [openib-general] Re: QoS RFC Service-ID: * For PathRecord: use the first 2 reserved fields whicg are 32bits each (component masks 0x1 and 0x2). Component mask 1 should be used to refer to the merged Service-ID field A new capability bit should describe the SM QoS support in the SA class port info. This approach provides an easy migration path for existing access layer and ULPs by not introducing a new attribute. This is OK but it's sort of a pain to have to query SA ClassPortInfo all the time. Do you have a plan for how to make this transparent to ULPs? (BTW something in your email client is really messing up the formatting of your message) - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] QoS RFC
Title: QoS RFC Eitan Lets assume for the moment (because youre presenting this under the OpenFabrics banner) that the capability you describe below will be hidden behind wire protocol agnostic API that can be used by IB HCA, Ethernet RNIC, etc. It is not clear from the description, but I assume the arbiter you refer to below allows for traffic classes where a) the bandwidth of the different flows within a particular class aggregate to a fixed bandwidth, and also b) where each flow within a class has a fixed bandwidth, e.g. where each flow corresponds to some type of fixed rate media stream. A QoS API should also needs have support for flows with large RTT (Round Trip Times). Large RTT flows typically require the sender to space the packets equally on the wire (referred to as pacing), with the spacing interval being dynamically computed based on the RTT. It is sufficient in this case to be able to set a pacing attribute through the API. Regards, Asgeir Eiriksson CTO Chelsio Communications From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Eitan Zahavi Sent: Tuesday, May 30, 2006 7:41 AM To: openib-general@openib.org Cc: Nimrod Gindi; Roland Dreier Subject: [openib-general] QoS RFC Hi All Please find the attached RFC describing how QoS policy support could be implemented in the OpenFabrics stack. Your comments are welcome. Eitan RFC: OpenFabrics Enhancements for QoS Support === Authors: . Eitan Zahavi [EMAIL PROTECTED] Date: May 2006. Revision: 0.1 Table of contents: 1. Overview 2. Architecture 3. Supported Policy 4. CMA functionality 5. IPoIB functionality 6. SDP functionality 7. SRP functionality 8. iSER functionality 9. OpenSM functionality 1. Overview Quality of Service requirements stem from the realization of I/O consolidation over IB network: As multiple applications and ULPs share the same fabric, means to control their use of the network resources are becoming a must. The basic need is to differentiate the service levels provided to different traffic flows. Such that a policy could be enforced and control each flow utilization of the fabric resources. IBTA specification defined several hardware features and management interfaces to support QoS: * Up to 15 Virtual Lanes (VL) could carry traffic in a non-blocking manner * Arbitration between traffic of different VL is performed by a 2 priority levels weighted round robin arbiter. The arbiter is programmable with a sequence of (VL, weight) pairs and maximal number of high priority credits to be processed before low priority is served * Packets carry class of service marking in the range 0 to 15 in their header SL field * Each switch can map the incoming packet by its SL to a particular output VL based on programmable table VL=SL-to-VL-MAP(in-port, out-port, SL) * The Subnet Administrator controls each communication flow parameters by providing them as a response to Path Record query The IB QoS features provide the means to implement a DiffServ like architecture. DiffServ architecture (IETF RFC2474 2475) is widely used today in highly dynamic fabrics. This proposal provides the detailed functional definition for the various software elements that are required to enable a DiffServ like architecture over the OpenFabrics software stack. cut to end of original email ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCH][iWARP] Added provider CM verbs andquery provider methods
Caitlin For the openib folk: keep in mind in the following that iWARP runs on top of TCP, and the TCP is offloaded so TOE enters the iWARP picture. I second the requirement that an iWARP RNIC needs to integrate with host configuration, reporting, and security mechanisms, and this is the approach taken in the Chelsio TOE patches that we have submitted. Standard tools such as netstat, ifconfig, work with the Chelsio TOE today, and there's nothing to prevent netfilter to work, etc. For those that are interested there is more information available at https://service.chelsio.com/open_toe and in particular in Chelsio_toe_arch.pdf I have to disagree that this means that the connection is necessarily setup by the host stack. The Chelsio 10GE NIC/TOE/iSCSI/iWARP products setup the connection on the card, but the setup includes an ASK host phase initiated by the card where the host can filter the connect request, and modify any of the initial TCP values chosen by the card, etc., and then respond with accept or reject. In general the iWARP connection manager needs to accommodate three possible TCP connection setup models in use today in iWARP devices: a) what you seem to be advocating, i.e. TCP connection setup on the host stack, b) what I brought up, i.e. offloaded TCP connection setup with an ASK phase, and c) what was brought up previously and that's full TCP connection setup offload. 'Asgeir From: Caitlin Bestler [EMAIL PROTECTED] To: Roland Dreier [EMAIL PROTECTED],Tom Tucker [EMAIL PROTECTED] CC: openib-general@openib.org Subject: RE: [openib-general] [PATCH][iWARP] Added provider CM verbs andquery provider methods Date: Thu, 25 Aug 2005 16:46:38 -0700 Device vendors would jump at the opportunity to have a stable interface with the host stack. Things like routing, protection from denial of service attacks, rules for logging and filtering connection requests and more all *should* be handled by the host stack. That's where the end user wants to control them, it's where the security code can be kept most current and most robust. It is also largely on packets that do not require offload optimization. But we also need time to ensure that the community understands this as giving the host stack control of an offload connection during connection establishment -- rather than as the offload device stealing the connection from the host stack. Moving the entire TCP connection logic to the offload device not only increases the work that the offload device must do, it reduces the auditability of the system and the user's control over their network activity. So the intent is not to evade the stack, rather it is to allow time for proper integration with host stack control. The tradeoffs are complex, and neither side fully understands the other's issues yet. We need to work together to determine how to provide the acceleration that our users want without sacrificing the OS provided security that they assume will not be sacrificed. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Roland Dreier Sent: Thursday, August 25, 2005 4:21 PM To: Tom Tucker Cc: openib-general@openib.org Subject: Re: [openib-general] [PATCH][iWARP] Added provider CM verbs and query provider methods Tom RNIC Verbs imply that the modify qp verb takes a handle to a Tom connection -- presumably a socket. This CAN'T be done on Tom Linux in any fashion that is acceptable to the netdev Tom crowd. SOOO we modeled this after DAPL. Trust me, I would Tom LOVE to be able to establish the connection using bind, Tom listen, etc..., query the Linux connection state and then Tom pass this down to the qp modify verb...but I can't. Let's not be too quick to say that this is impossible. I think we should work with the Linux networking community and come up with the right answer, and not accept a bad solution just because it lets us go around the networking people. Has there been any real discussion of this on netdev? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCH][iWARP] Added provider CM verbs andquery provider methods
Roland We're planning to go back with a new submission which addresses the concerns that were directly relevant to the patch itself. In the process, we'll be porting the patch to 2.6.14. A couple of comments: If one were to look at the patch in its current form, you'd find that it is already quite minimal compared to the changes needed for 10GE TCP/IP alternatives. The architecture that we propose also accommodates different TOE approaches, e.g. different connection setup models, etc. We currently have the proposed architecture running on Linux in conjunction with a regular NIC, iWARP RNIC, and iSCSI HBA. Finally, with the new submission, we're hoping to get a more constructive dialogue going, which focuses on the patch itself, because it is clear that there is user interest in the technology, and Linux support would be beneficial to all parties. 'Asgeir -Original Message- From: Roland Dreier [mailto:[EMAIL PROTECTED] Sent: Monday, August 29, 2005 9:24 AM To: Asgeir Eiriksson Cc: openib-general@openib.org Subject: Re: [openib-general] [PATCH][iWARP] Added provider CM verbs andquery provider methods Asgeir ...this is the approach taken in the Chelsio TOE patches Asgeir that we have submitted. What are your plans for these patches? I am not subscribed to netdev, but from reading the archives, it seems that your most recent submission was rejected quite strongly. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
