[openib-general] RE: OpenSM Work
Title: RE: OpenSM Work Hi Hal, Please see my responses below. As Mellanox moves to work on OpenIB Gen2 stack, we have assigned Yael to work on merging OpenSM 1.8.0 (which released based on gen1) into the gen2 stack. I too am working on small pieces of this. [EZ] I think it does not make much sense to work in parallel as the changes spans many files. What are the changes you work on? She has started to work on the merge to ensure that fixes done by you and Shahar on the gen2 trunk will not be lost. The mode of work we suggest is that she will work offline. Not sure by what you mean by offline here. [EZ] Offline means she will do the entire merge and then commit. I propose she will commit the changes into a branch and then you can review it and do the merge to the main trunk yourself. When the merge will be completed a side branch will be opened under: https://openib.org/svn/gen2/branches/osm_1_8_0 and will made available for review and testing before merge into the main trunk. I would prefer small patches rather than a large merge if that were possible. [EZ] I am afraid this is impossible due to the large difference in code. The changes affect many files and will require heavy testing to make sure they did not break anything. The testing will be done at the end of the merge work. We could however, let her commit small changes - one at a time - but that branch will be useless. Once all this is done, Yael will work on multiple new features including faster route time, PKey manager, MKey manager, and QoS. She will do so on branches off the main trunk - for each feature. OK. Sounds good. In parallel, Liran who owns the OpenSM verification will enhance osmtest and other testing utilities to achieve better test coverage of SM handover, SL2VL, VLArb and PKey. Any new feature will get covered by new tests. OK. I myself will work on making sure the IB management simulator is well integrated with the stack What do you mean by stack here ? [EZ] I mean OpenIB and the available simulator based tests as well as new tests can be run daily. We do have some issues with respect to the current osm tree: 1. All header files were moved from their relative location under the opensm, complib, iba directories and placed under the include directory. Although this seems reasonable for a install tree - it is not very common for development trees. Normally I would expect the Makefile.am of each sub directory of the osm project to define which header files are to be installed into the $prefix/include dir. We will revert that hierarchy change in our merged branch. Are you expecting the reverted hierarchy to make it back to the trunk ? [EZ] Yes. Explained why before. If you need examples for how this is done in many other GNU projects we can provide it. 2. osmtest was just introduced back into the osm tree. I think osmtest should be placed under a test tree where all the tests of the ULPs core etc will be located. I would expect a location like: https://openib.org/svn/gen2/trunk/test/userspace/management/osm It can be moved when it is agreed on its location. There are other places being proposed for this (talk to Amit). [EZ] OK 3. osmtest needs cleanup from VAPI stuff - we should let Liran who is the owner of this code development a clear AR to clean it up. Yes. Can he send patches for this ? [EZ] Yes he will 4. For some reason I saw that you have added Voltaire copyright to the osmtest code. I do not think it makes sense as no work was done on this code by a Voltaire developer. Or I might be wrong? There was some minor work done in terms of OpenIB. I removed this. Needless to say the 1.8.0 version of OpenSM brings with it a long set of bug fixes and enhancements. Once the OpenSM work is completed, will OpenSM development by Mellanox be done incrementally and in the open rather than drops ? Will patches be suppplied ? [EZ] Yes. After the merge is done all enhancements described above will be done on OpenIB. Assuming we resume our maintainer position and are able to commit directly. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes
Quoting r. yhlu [EMAIL PROTECTED]: Subject: Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes Roland, In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB driver in Kernel can not be loaded. Are you using the latest firmware on the HCA card? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] IPoIB -- connected mode update
Attached is an udpated draft (will be posting to internet drafts after the current ietf ends) for ipoib-connected mode based on the discussions on ipoib wg, openib (IB on Linux), and other communications. Two threads that saw good discussion are given below. I believe the attached updated draft captures all the discussions. Please comment. http://openib.org/pipermail/openib-general/2005-May/006751.html http://www1.ietf.org/mail-archive/web/ipoverib/current/msg01212.html thanks, Vivek IP over InfiniBand: Connected Mode Abstract This document specifies a method for transmitting IPv4/IPv6 packets and address resolution over the connected modes of InfiniBand. Table of Contents 1.0 Introduction 2.0 IPoIB-connected mode 2.1 Multicasting 2.2 Outline of Address Resolution 2.3 Outline of Connection Setup 3.0 Address Resolution 3.1 Link-layer Address 3.2 IB Connection Setup 3.3 Service-ID 4.0 Frame Format 5.0 Maximum Transmission Unit 5.1 Per-Connection MTU 6.0 IPoIB-CM Considerations 6.1 A Cautionary Note on IPoIB-RC 7.0 Security Considerations 8.0 IANA Considerations 9.0 References 1.0 Introduction The InfiniBand specification [IB_ARCH] can be found at www.infinibandta.org. The document [IPoIB_ARCH] provides a short overview of InfiniBand architecture along with consideration for specifying IP over InfiniBand networks. The InfiniBand architecture (IBA) defines multiple modes of transports. Of these the unreliable datagram (UD) transport method best matches the needs of IP. IP over InfiniBand (IPoIB) over UD is described in [IPoIB_UD]. This document describes IP transmission over the connected modes of IBA. IBA defines two connected modes: 1. Reliable Connected (RC) 2. Unreliable Connected (UC) As is evident from the nomenclature, the two modes differ mainly in providing reliability of data delivery across the connection. This document applies equally to both the connected modes. IPoIB over these two modes is referred to as IPoIB-CM (connected mode) in this document. For clarity IPoIB over the unreliable datagram mode, as described in [IPoIB_UD] is referred to as IPoIB-UD. IBA requires that all Host Channel Adapters (HCAs) support the reliable and unreliable connected modes [IB_ARCH]. It is optional for Target Channel Adapters (TCAs) to support the connected modes. The connected modes offer link MTUs of up to 2^31 octets in length. Thus the use of connected modes can offer significant benefits by supporting reasonably large MTUs. The datagram modes of InfiniBand Architecture (IBA) are limited to 4096 octets. Reliability is also enhanced if the underlying feature of automatic path migration supported by the connected modes is utilized. The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in RFC 2119. 2.0 IPoIB-connected mode Every IPoIB implementation MUST support IPoIB-UD. The IPoIB-CM support is OPTIONAL. This document extensively refers to [IPoIB_UD] and extends IPoIB description given in [IPoIB_UD] to IPoIB-CM. Therefore, only additional requirements or enhancements needed to enable IPoIB- CM are described. The IP encapsulation, default MTU, link layer address format and the IPv6 stateless autoconfiguration mechanism apply to IPoIB-CM exactly as described in [IPoIB_UD]. 2.1 Multicasting The connected modes of IBA define a non-broadcast, multiple access network. The connected modes of IBA do not support multicasting though every node can communicate with every other node if desired. This requires that multicasting be emulated in some form by the network. However, in the case of an InfiniBand network, instead of an emulation, an unreliable datagram (UD) queue pair (QP) can be used to support multicasting while the connected mode QP is used for unicast traffic. Since every IPoIB implementation is required to support the UD mode, every implementation supporting IPoIB-CM will be able to utilize the coexisting IPoIB-UD QP for all broadcast/multicast communications. Multicast mapping, transmission and reception of multicast packets and multicast routing MUST use the IPoIB-UD QP associated
[openib-general] 32b openib applications on 64b kernel
Does the openib code support a 32b app using user-mode IB verbs on a 64b distro/kernel? IE: The app was compiled on a 32b distro/kernel, then run on the 64b distro/kernel. Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] OpenMPI question
Will the various MPI implementations be ported to the openIB user verbs API? Or uDAPL? From reading the FAQ it appears that MVAPICH2 will remain on udapl. I'm curious about Open-MPI and LA-MPI? Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: OpenSM Work
Title: RE: [openib-general] RE: OpenSM Work Hi Matt, The offline work is only expected for the task of merging of the current 1.8.0 release into OpenIB. It should be completed by end of next week. It would involve the following parts: 1. Merging changes done by Hal/Shahar on the 1.6.1 and 1.7.0 into the 1.8.0 code. 2. Moving headers back into place and adding Makefile.am install directives to move the headers during install to the prefix/include dir 3. Adding osm_vendor_sa_api.h and accompany implementation for that API. 4. Adding Simulator vendor After that point we wish to continue OpenSM development on the OpenIB repository. To be able to do that we need to open a branch for each new feature we develop, qualify it and then we can provide diffs to the main thread so everybody can review it before the merge. Actually with the amount of work we put already into OpenSM, once we move into gen2 environment we should become the active maintainers of this code. The development work plan for OpenSM was posted more then once to the OpenIB mailing list even before we started merging with gen2 (attached to this mail). We will appreciate feedback for our plans and will adjust accordingly. So far I did not get any response for my previous mails describing our plans. I hope this will change. Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -Original Message- From: Matt Leininger [mailto:[EMAIL PROTECTED]] Sent: Thursday, August 04, 2005 4:26 PM To: Eitan Zahavi Cc: openib-general@openib.org; 'Hal Rosenstock' Subject: Re: [openib-general] RE: OpenSM Work On Thu, 2005-08-04 at 09:09 +0300, Eitan Zahavi wrote: The mode of work we suggest is that she will work offline. Not sure by what you mean by offline here. [EZ] Offline means she will do the entire merge and then commit. I propose she will commit the changes into a branch and then you can review it and do the merge to the main trunk yourself. Eitan, I'm glad to see the continued interest in OpenSM. Thanks for the help. Please submit OpenSM changes as patches to the mail list so that Hal and others in the community can review them. No sense in doing a bunch of work until we know what things you are trying to add. Start with header files and work out from there. My understanding of your offline is that it is closed development followed by open release. That's not how OpenIB works. Please submit the patches to the list so everyone can follow and commit on the OpenSM code changes. Thanks, - Matt ---BeginMessage--- Title: OpenSM work Hi All, FYI: Mellanox is focusing on the following items on OpenSM development for the last few weeks: 1. Stability testing over the IB management simulator: a. Randomly pick bad links with high packet drop statistics - success is SUBNET UP b. Route using up/down algorithm - success is no credit loops 2. Semi-static LID assignment: a. Developed an interface for persistent storage of arbitrary data. The goal is to enable further development of LDAP (ala Troy's request) or SQL module. Please see osm_db.h attached osm_db.h b. Developed file based implementation for osm_db.h c. Modify osm_lid_mgr (lid assignment algorithm) to use the LIDs stored in the persistent storage. Handle all cases of bad file and new LIDs on the fabric. The -r flag now lets OpenSM overwrite the known data. Persistent Guid to LIDs data is kept even if the GUID disappears for a while. The code also handles LID assignment for LMC 0 in a way better then the previous algorithm: It used to assign 2^LMC LIDs for every port - even for switches port 0. Now it will only preserve 1 LID for switch port 0. 3. Irresponsive port: a. The phenomenon is: A port does not respond to the SM during the discovery stage. OpenSM can not obtain enough data about the port and thus it does not appear in the final database. Since OpenSM uses light sweeps when there is no "change detected" it will not query the port until either a switch sets its "change bit" or send a trap. So that irresponsive port will never be polled again if there is no heavy sweep. b. The solution: i. During discovery track ports (physical ports) that have their logical link state != DOWN but the port on the other side of the link is not known to the SM. ii. During light sweep: not only scan the switches "change bit" but also test to see if the port on the other side on these ports (from i) is responding. If it does - issue a heavy sweep. 4. Head of Queue Life: a. Problem: In cases of PCI hardware failure HCAs can not complete RDMA requests and loose all credits from their input ports (in other words: their input buffers are filled). So they create back pressure on the fabric. b. Solution: use a fast head of queue time limit on every switch port that drives an HCA. 5. SA queries stress testing: a. We are exploring max performance
Re: [openib-general] OpenMPI question
On Thu, 2005-08-04 at 09:52, Steve Wise wrote: Will the various MPI implementations be ported to the openIB user verbs API? Or uDAPL? From reading the FAQ it appears that MVAPICH2 will remain on udapl. OSU MVAPICH2 is being ported now to OpenIB uverbs. It's Intel's MPI which is over uDAPL. I'm curious about Open-MPI and LA-MPI? OpenMPI is working with OpenIB uverbs now (in point to point mode) but has not yet been released. Don't know about LA-MPI. -- Hal Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] Move InfiniBand .h files
Grant Any thoughts on renaming drivers/infiniband to drivers/rdma Grant at the same time? Grant If you are going to churn...don't be shy about it :^) Well, I'd rather avoid churn for purely political reasons. The main point of my proposal is to move the includes from drivers/ to include/, but while we're at it me might as well pick a more neutral directory name. Moving drivers/infiniband to drivers/rdma has no technical merit right now, so I'd rather wait and see how it makes sense to organize the code we end up with. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: mthca and LinuxBIOS
yhlu i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y I didn't get any yhlu more debug info, is that depend other setting? It shouldn't depend on anything. mthca_dbg() gets turned into dev_dbg(), which just does dev_printk(KERN_DEBUG,...). Perhaps you have to change your console level to see KERN_DEBUG messages? Since you're getting to the call to mthca_init_qp_table(), there are mthca_dbg() calls that you should definitely be getting output from. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] Move InfiniBand .h files
Roland Also, drivers/infiniband/include doesn't get put into the Roland /lib/modules/ver/build directory, Arjan that is a symlink not a directory, and a symlink to the Arjan full source... Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their support kernel builds package. And if I use an external build directory (ie O=) then the symlink just points to my external build directory, which doesn't include the source to drivers/, just links to include/ - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] Move InfiniBand .h files
On Thu, Aug 04, 2005 at 11:57:55AM -0700, Roland Dreier wrote: Roland Also, drivers/infiniband/include doesn't get put into the Roland /lib/modules/ver/build directory, Arjan that is a symlink not a directory, and a symlink to the Arjan full source... Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their support kernel builds package. And if I use an external build directory (ie O=) then the symlink just points to my external build directory, which doesn't include the source to drivers/, just links to include/ If the external module uses a Kbuild file as explained in Documentation/kbuild/makefiles.txt and then uses both O= and M= when compiling the module there is no issue. With respect to moving the .h files - please do so. drivers/infiniband should only include header used in that same directory. Not header files potentially uased by fs/. Sam ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: mthca and LinuxBIOS
ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005) ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (:04:00.0) ib_mthca :04:00.0: FW version 000400060002, max commands 64 ib_mthca :04:00.0: FW size 6143 KB (start fcefa0, end fcefff) ib_mthca :04:00.0: HCA memory size 262143 KB (start fce000, end fcefff) ib_mthca :04:00.0: Max QPs: 16777216, reserved QPs: 1024, entry size: 256 ib_mthca :04:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64 ib_mthca :04:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64 ib_mthca :04:00.0: reserved MPTs: 16, reserved MTTs: 16 ib_mthca :04:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1 ib_mthca :04:00.0: Max QP/MCG: 16777216, reserved MGMs: 0 ib_mthca :04:00.0: Flags: 00370347 ib_mthca :04:00.0: profile[ 0]--10/20 @ 0x fce000 (size 0x 400) ib_mthca :04:00.0: profile[ 1]-- 0/16 @ 0x fce400 (size 0x 100) ib_mthca :04:00.0: profile[ 2]-- 7/18 @ 0x fce500 (size 0x 80) ib_mthca :04:00.0: profile[ 3]-- 9/17 @ 0x fce580 (size 0x 80) ib_mthca :04:00.0: profile[ 4]-- 3/16 @ 0x fce600 (size 0x 40) ib_mthca :04:00.0: profile[ 5]-- 4/16 @ 0x fce640 (size 0x 20) ib_mthca :04:00.0: profile[ 6]--12/15 @ 0x fce660 (size 0x 10) ib_mthca :04:00.0: profile[ 7]-- 8/13 @ 0x fce670 (size 0x 8) ib_mthca :04:00.0: profile[ 8]--11/11 @ 0x fce678 (size 0x 1) ib_mthca :04:00.0: profile[ 9]-- 6/ 5 @ 0x fce679 (size 0x 800) ib_mthca :04:00.0: HCA memory: allocated 106050 KB/256000 KB (149950 KB free) ib_mthca :04:00.0: Allocated EQ 1 with 65536 entries ib_mthca :04:00.0: Allocated EQ 2 with 128 entries ib_mthca :04:00.0: Allocated EQ 3 with 128 entries ib_mthca :04:00.0: Setting mask 000f43fe for eqn 2 ib_mthca :04:00.0: Setting mask 0400 for eqn 3 ib_mthca :04:00.0: NOP command IRQ test passed --stuck 30s ib_mthca :04:00.0: Failed to initialize queue pair table, aborting. ib_mthca :04:00.0: Clearing mask 000f43fe for eqn 2 ib_mthca :04:00.0: Clearing mask 0400 for eqn 3 ib_mthca: probe of :04:00.0 failed with error -16 On 8/4/05, Roland Dreier [EMAIL PROTECTED] wrote: yhlu i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y I didn't get any yhlu more debug info, is that depend other setting? It shouldn't depend on anything. mthca_dbg() gets turned into dev_dbg(), which just does dev_printk(KERN_DEBUG,...). Perhaps you have to change your console level to see KERN_DEBUG messages? Since you're getting to the call to mthca_init_qp_table(), there are mthca_dbg() calls that you should definitely be getting output from. - R. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [RFC] Move InfiniBand .h files
On Thu, 2005-08-04 at 11:26 -0700, Grant Grundler wrote: On Thu, Aug 04, 2005 at 07:53:58PM +0200, Arjan van de Ven wrote: On Thu, 2005-08-04 at 10:32 -0700, Roland Dreier wrote: I would like to get people's reactions to moving the InfiniBand .h files from their current location in drivers/infiniband/include/ to include/linux/rdma/. If we agree that this is a good idea then I'll push this change as soon as 2.6.14 starts. please only put userspace clean headers here; the rest is more or less private headers for your subsystem. Sorry...this smells like a rathole...but does this mean linus agrees the kernel subsystems should export headers suitable for both user space and kernel driver modules? Historical, I thought glibc and other user space libs were expected to maintain their own set of header files. Maybe I'm just confused... there is a definite requirement for the kernel to expose SOME things to userspace. Well for SOMETHING to expose them. Right now most distros ship a hacked up version of the kernel headers (eg removed of all the kernel specific stuff and all the gpl inline code etc). A good part of making such an external project possible is to make a clean separation between userspace shared stuff and pure kernel internals. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] Move InfiniBand .h files
On Thu, 2005-08-04 at 11:57 -0700, Roland Dreier wrote: Roland Also, drivers/infiniband/include doesn't get put into the Roland /lib/modules/ver/build directory, Arjan that is a symlink not a directory, and a symlink to the Arjan full source... Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their support kernel builds package. so what makes you think they will ship include/infiniband ? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] Move InfiniBand .h files
Sam Ravnborg wrote: On Thu, Aug 04, 2005 at 11:57:55AM -0700, Roland Dreier wrote: Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their support kernel builds package. And if I use an external build directory (ie O=) then the symlink just points to my external build directory, which doesn't include the source to drivers/, just links to include/ If the external module uses a Kbuild file as explained in Documentation/kbuild/makefiles.txt and then uses both O= and M= when compiling the module there is no issue. With respect to moving the .h files - please do so. drivers/infiniband should only include header used in that same directory. Not header files potentially uased by fs/. I think Roland was talking about the case where the running kernel was built with O=, in which case the /lib/modules.../build symlink points to the build directory rather than the original source tree. Does Kbuild handle this case properly? Chris ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] Move InfiniBand .h files
On Thu, Aug 04, 2005 at 01:54:56PM -0600, Christopher Friesen wrote: Sam Ravnborg wrote: On Thu, Aug 04, 2005 at 11:57:55AM -0700, Roland Dreier wrote: Sorry, I was too terse about the problem. You're right, but typical distros don't ship full kernel source in their support kernel builds package. And if I use an external build directory (ie O=) then the symlink just points to my external build directory, which doesn't include the source to drivers/, just links to include/ If the external module uses a Kbuild file as explained in Documentation/kbuild/makefiles.txt and then uses both O= and M= when compiling the module there is no issue. With respect to moving the .h files - please do so. drivers/infiniband should only include header used in that same directory. Not header files potentially uased by fs/. I think Roland was talking about the case where the running kernel was built with O=, in which case the /lib/modules.../build symlink points to the build directory rather than the original source tree. Does Kbuild handle this case properly? Yes it does. /lib/modules/.../ contains two symlinks these days: build - always point to the directory containing the output of the build source - always point to the kernel source In the 'make' case where the kernel is built without using O= they point to the same directory. In the 'make O=' case they point to different directories. SUSE does ship with a make O= build kernel these days. Fedora IIRC has done an ugly hack and just copied over a number of files so a compile works in most cases - but then also use both symlink. It has never been easier to build a module if the target is only the running kernel. Only when you adds backwards compatibility it gets messy :-( Sam ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCH 1/2] kDAPL: remove inline functions
On Thu, 4 Aug 2005, Tom Duffy wrote: On Thu, 2005-08-04 at 20:30 +0300, Or Gerlitz wrote: This is almost exactly what is there now. Not really a change. I don't like the multiple indirections to get to the function table, but this can be simplified later. Indeed, it can be simplified a little, but why not having this simplification in the kdapl registry level (dat) as it is in ib_core calling ib_verbs and libibverbs calling libmthca, while the verbs consumer does not have to go into those indirections at all but rather just call a well understood/defined api? are you suggesting to apply such a chance also to the verbs? I don't understand what you mean? What change would happen in verbs? I think Or is asking if similar changes should be made to the verbs (i.e. should ib_alloc_pd() be remove and replaced in each verb's user by device-alloc_pd(..), etc.). Is that right Or? In kDAPL, the inline dat_* functions just perform a function call. In the case of the verbs, there are some operations performed by the verbs core in addition to calling the function. For example, ib_alloc_pd() intiailizes the ib_pd fields in addition to calling the device specific alloc_pd function. Due to the additional functionality, I don't think this convention would extend to the IB verbs. In terms of kDAPL, I share Or's concern that this will make the API harder for consumers to use. Picture someone who wants to use the kDAPL API for the first time. Today that person would scan the kDAPL header and see all of the functions available and get a pretty clear picture of what each functions parameters are for just by looking at the parameter names. We'd loose that documentation with this change. With respect to the struct file_operations analogy, is a struct file_operations embedded in any other structure besides struct file? In kDAPL, the function table is embedded in several different objects. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general][kdapl]: vmalloc instead of kmalloc
On Thu, 4 Aug 2005, Guy German wrote: James, I see what you mean. The allocation of the event vector is derived from evd-qlen. In DTO ev'd, however, qlen is also the parameter passed to ib_create_cq. Since we don't want to limit DAPL consumers to an unnecessary small completion queue size, maybe we could differentiate between DTO supporting evd's and CONN evd's, when allocating the events vector. if evd supports CONN only, leave it : event = kmalloc(evd-qlen * sizeof *event) (Relying on the consumer he knows what he is doing) if evd is DTO only : don't allocate an event buffer, at all if evd supports both : event = kmalloc(DEFAULT_4_CONN * sizeof *event) And dynamically add additional events up to qlen as needed? if DEFAULT_4_CONN=256, that's a 3 pages allocation. How does that sound to you ? I'd prefer that the EVDs were uniform. I would worry about bugs otherwise. The eventual solution has to support qlen generated events (connection request, connection, disconnect, software) if those event types are supported by the EVD (even if the EVD is being used for both generated events and DTOs). ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: some questions/comments on gen2 udapl code
Or, We need to have these discussions on openib-general. -Original Message- From: Or Gerlitz [mailto:[EMAIL PROTECTED] Sent: Thursday, August 04, 2005 6:55 AM To: Davis, Arlin R Cc: James Lentini Subject: RE: some questions/comments on gen2 udapl code Arlin writes: I take this back. With the current verbs event mechanisms, a dedicated thread for uAT, uCM, and uCQ is the only way to process multi-threaded application requests. Let's say an application opens the device and then kicks off separate threads to connect and send to many different endpoints. Each thread will be waiting on the same event mechanisms. I see no good way to multiplex properly outside of a dedicated processing thread. If the verbs AT, CM, and CQ event mechanisms change then we can re-visit. I am not sure there's is no good way - but i guess not sure is not enough, i need to send a patch that allows for it (ie work under multi-threaded app (eg udapltest) with each thread having diff conn/dto/cr evd)... Patch would be great. I moved to a dedicated CQ thread approach after struggling with the many multi-threaded issues with direct waits. See comments in dapls_ib_wait_object_wait() in dapl_ib_cq.c. The thing is that the current implementation is not optimal in delivering CQ interrupts latency, since there's one extra context switch. Also the udapl provider I agree that this is not optimal and we want the CQ_WAIT_OBJECT mapped directly to a CQ fd to avoid the extra context switch. But with one CQ FD per device how do you optimally multiplex the user's CQ context across multiple waiters without a dedicated processing thread? It seems just as inefficient to have 4 waiters on different CQ's all waking up, processing an event that may or may not be theirs. How do you deliver the event that was not yours? What if the event was directed to a thread that already woke and went back to sleep? Do you wake them all up again? This is why I went back to the dedicated CQ thread. We need some help from verbs to make this uDAPL CQ mapping work: Either support multiple CQ FD's, one per CQ. or modify call ibv_get_cq_event() and allow the user to supply the specific ibv_cq event to get, not just the first one on the event queue. library opens 3 dedicated threads (uat/ucm/ucq) and for getting async event we will probably add uasync thread - which I can work on rolling up the uat/ucm/uasync processing into one thread. -arlin would be better if we can avoid doing (opening so many threads). Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] ib_memory_register() equivalent in gen2
Hi, I'm looking for the memory registration function which takes a vaddr and and length, not an array of physical buffers. In gen1 this call is ib_memory_register(). The closest thing I can find is the reg_user_mr stuff which is used by uverbs. Could someone please tell me if req_user_mr is the right place to start? Thanks, Paul ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] SDP and uCM.
At the end of this week I will be leaving Cisco and will no longer have the time, or access to the equipment, needed to maintain the SDP or uCM code. It would seem ideal for Sean Hefty to take over the uCM code if he has the time and desire. I also think that Tom Duffy would be the right person to maintain the SDP code, unless he doesn't think he has the time. In which case Michael Tsirkin knows the code very well and would hopefully consider taking over the code. Thank you all. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: SDP and uCM.
On Thu, 2005-08-04 at 14:26 -0700, Libor Michalek wrote: At the end of this week I will be leaving Cisco and will no longer have the time, or access to the equipment, needed to maintain the SDP or uCM code. It would seem ideal for Sean Hefty to take over the uCM code if he has the time and desire. I also think that Tom Duffy would be the right person to maintain the SDP code, unless he doesn't think he has the time. In which case Michael Tsirkin knows the code very well and would hopefully consider taking over the code. Libor, do you plan on checking anything else into SDP before you give over maintainership? -tduffy signature.asc Description: This is a digitally signed message part ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] SDP and uCM.
or uCM code. It would seem ideal for Sean Hefty to take over the uCM I can maintain the uCM code along with the kCM. Good luck with your new endeavor! - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: SDP and uCM.
On Thu, Aug 04, 2005 at 02:46:47PM -0700, Tom Duffy wrote: On Thu, 2005-08-04 at 14:26 -0700, Libor Michalek wrote: At the end of this week I will be leaving Cisco and will no longer have the time, or access to the equipment, needed to maintain the SDP or uCM code. It would seem ideal for Sean Hefty to take over the uCM code if he has the time and desire. I also think that Tom Duffy would be the right person to maintain the SDP code, unless he doesn't think he has the time. In which case Michael Tsirkin knows the code very well and would hopefully consider taking over the code. Libor, do you plan on checking anything else into SDP before you give over maintainership? No. I don't have any private patches to commit, and it does not look like I'll have time to look at and test all the patches that have been posted. Sorry. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH updated] sdp: cancel read with no iocb
On Mon, Aug 01, 2005 at 04:14:45PM +0300, Michael S. Tsirkin wrote: Quoting r. Michael S. Tsirkin [EMAIL PROTECTED]: Subject: sdp: cancel read with no iocb Libor, I'm seeing these messages: ib_sdp WARN: Cancel read with no IOCB. 2:0:0005 It seems that this warning is printed in a legal state where a deferred iocb is canceled. Shouldnt this sdp_warn be replaced with sdp_dbg_ctrl? You're right, this should have been a sdp_dbg_ctrl all along, since it's even possible, although much less likely, when the completion is performed in irq context. The write cancel in sdp_send.c should be changed as well. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] remove in_atomic
On Mon, Aug 01, 2005 at 09:35:29AM +0300, Michael S. Tsirkin wrote: in_atomic isnt a reliable way to check that we are in an atomic context. Just schedule work, always, since most cq polling is currently done under a spinlock, anyway. I agree this patch is correct, but did you see any performance change? With it applied I'm seeing a lot more variance in performance from run to run. I believe the differences look to do with timing and how frequently we are using source rdma advertisement vs. the sink. I believe the slow start algorithm for src_avails needs improvment. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [iSER]How to use the iSER with the UNH iSCSI
Ian, Currently the UNH iSCSI doesn’t support the Datamover API which is a new API defined in IETF and enable iSCSI to run over offload technologies such as iSER In addition the iSER code that is in OpenIB covers the Initiator side The Target code is (and being) integrated into few commercial products, or can be provided under some licensing There are few that intend to enable the datamover API in the UNH iSCSI and integrate it with iSER, they would be happy to see more helping hands, if you are interested I can hook you up with them Yaron -Original Message- From: [EMAIL PROTECTED] [mailto:openib-general- [EMAIL PROTECTED] On Behalf Of Ian Jiang Sent: Thursday, August 04, 2005 3:10 AM To: openib-general@openib.org Subject: [openib-general] [iSER]How to use the iSER with the UNH iSCSI Hi, everybody! Thanks for all the replis to my How to get the dat_headers_1_1.tgz! I downloaded the dapl_beta2.06.tgz as Itamar told me. And I made some modification to the iSER to use it on the x86_64 platform. I got through the compiling finally, but here is another question: How to use the iSER with the UNH iSCSI? I have the UNH iSCSI running on my system at present. Need I modify it and reinstall? And I'm not sure if the dapl_beta2.06 has to be installed to run the iSER. In fact, I did not compile or install the dapl before installing the iSER. Any suggestion is appriciated! Ian Jiang [EMAIL PROTECTED] Computer Architecture Laboratory Institute of Computing Technology Chinese Academy of Sciences Beijing,P.R.China Zip code: 100080 Tel: +86-10-62564394(office) _ 免费下载 MSN Explorer: http://explorer.msn.com/lccn ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: mthca and LinuxBIOS
Hmm, that output all looks fine. Can you run with the patch below to
see exactly where the QP table initialization fails? (I haven't
actually compiled this patch so you may have to fix a typo or two)
I'm guessing that the CONF_SPECIAL_QP command is failing, but let's
make sure.
Thanks,
Roland
diff --git a/drivers/infiniband/hw/mthca/mthca_qp.c
b/drivers/infiniband/hw/mthca/mthca_qp.c
--- a/drivers/infiniband/hw/mthca/mthca_qp.c
+++ b/drivers/infiniband/hw/mthca/mthca_qp.c
@@ -2214,13 +2214,16 @@ int __devinit mthca_init_qp_table(struct
(1 24) - 1,
dev-qp_table.sqp_start +
MTHCA_MAX_PORTS * 2);
- if (err)
+ if (err) {
+ mthca_err(dev, mthca_init_qp_table: mthca_alloc_init failed
(%d)\n, err);
return err;
+ }
err = mthca_array_init(dev-qp_table.qp,
dev-limits.num_qps);
if (err) {
mthca_alloc_cleanup(dev-qp_table.alloc);
+ mthca_err(dev, mthca_init_qp_table: mthca_array_init failed
(%d)\n, err);
return err;
}
@@ -2228,8 +2231,10 @@ int __devinit mthca_init_qp_table(struct
err = mthca_CONF_SPECIAL_QP(dev, i ? IB_QPT_GSI : IB_QPT_SMI,
dev-qp_table.sqp_start + i * 2,
status);
- if (err)
+ if (err) {
+ mthca_err(dev, mthca_init_qp_table:
mthca_CONF_SPECIAL_QP failed for %d/%d (%d)\n, i, dev-qp_table.sqp_start + i
* 2, err);
goto err_out;
+ }
if (status) {
mthca_warn(dev, CONF_SPECIAL_QP returned
status %02x, aborting.\n,
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] ib_memory_register() equivalent in gen2
pauln Hi, I'm looking for the memory registration function which pauln takes a vaddr and and length, not an array of physical pauln buffers. In gen1 this call is ib_memory_register(). The pauln closest thing I can find is the reg_user_mr stuff which is pauln used by uverbs. Could someone please tell me if pauln req_user_mr is the right place to start? No, reg_user_mr is used to register userspace memory. There is no memory registration function for kernel consumers that takes a virtual address and a length. This is intentional, because there is not a general way to translate a kernel virtual address to a bus address that can be passed to a device. In fact, not every kernel virtual address may be used for DMA, and not every piece of DMA-able memory has a kernel virtual address. Every kernel consumer needs to handle the mapping of its memory to bus addresses (note -- _not_ physical addresses). - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
