Re: [ewg] OFED1.4.1 SLES11 backport builds with sles11 rc4 kernel
On Thu, 26 Feb 2009 16:25:37 -0800 Jeff Becker wrote: > I had trouble building with-ehca-mod because include/asm-powerpc is no > longer there in 2.6.27 and later: e.g., for > drivers/infiniband/hw/ehca/ehca_main.c: > > In file included from > /home/becker/OFED_NFS_RDMA/sles11rc4-build/drivers/infiniband/hw/ehca/ipz_pt_fn.h:51, > from > /home/becker/OFED_NFS_RDMA/sles11rc4-build/drivers/infiniband/hw/ehca/ehca_classes.h:64, > from > /home/becker/OFED_NFS_RDMA/sles11rc4-build/drivers/infiniband/hw/ehca/ehca_main.c:49: > /home/becker/OFED_NFS_RDMA/sles11rc4-build/drivers/infiniband/hw/ehca/ehca_tools.h:62:26: > error: asm/abs_addr.h: No such file or directory > /home/becker/OFED_NFS_RDMA/sles11rc4-build/drivers/infiniband/hw/ehca/ehca_tools.h:63:25: > error: asm/ibmebus.h: No such file or directory > /home/becker/OFED_NFS_RDMA/sles11rc4-build/drivers/infiniband/hw/ehca/ehca_tools.h:66:24: > error: asm/hvcall.h: No such file or directory > > Any advice? Thanks. > > -jeff Hi Jeff, The asm directory is now located in arch/powerpc/include. I have been testing the daily builds of OFED 1.4.1 against a 2.6.27 vanilla kernel on powerpc (the ehca driver only works on powerpc platforms). That worked fine so far. Can you please send me the package you try to build on SLES11 so that I can recreate this here? Anyway, thanks for looking into the SLES11 backports, I will be happy to test them with ehca when they are integrated into the daily builds. Regards, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Please pull libehca.git
Hi Vlad, please pull the libehca.git to get two fixes for OFED-1.4.1. The following changes since commit e0c2d7e8ee2aa5dd3f3511270521fb0c206167c6: Joachim Fenkes (1): Make req_notify_cq() MMIO stores safe in 32bit mode are available in the git repository at: git://git.openfabrics.org/~hnguyen/libehca.git master Alexander Schmidt (2): libehca: whitespace / style cleanup improve error handling in destroy_cq() src/ehca_uabiver5.c |7 +--- src/ehca_uabiver6.c | 10 + src/ehca_uabiver7.c | 10 + src/ehca_uclasses.h |2 +- src/ehca_umain.c| 112 +-- 5 files changed, 59 insertions(+), 82 deletions(-) ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Delaying next Monday OFED meeting
On Thu, 5 Mar 2009 14:23:02 +0200 "Tziporet Koren" wrote: > Hello, > Due to Purim holiday in Israel I wish to delay the next Monday OFED > meeting. > We can do it next week on Thursday (12 March) 9am PST or delay to a week > after on Monday (March 16 ) 9am PST > > Can you reply with your availability? > > Sorry for this inconvenient. > > Tziporet Hi Tziporet, I am available on both dates. Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] OFED-1.4.1 daily: build error in ib-bonding
Hi all, with the latest daily build (OFED-1.4.1-20090305-0600), I run into a build problem with ib-bonding, running on SLES10 SP2 on powerpc. The last daily build I tested (OFED-1.4-20090208-0600) worked fine, it contained ib-bonding-0.9.0-36 (new version is ib-bonding-0.9.0-38). Is anyone else seeing this? Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.4571 + umask 022 + cd /var/tmp/OFED_topdir/BUILD + /bin/rm -rf /var/tmp/ib-bonding-0.9.0-root ++ dirname /var/tmp/ib-bonding-0.9.0-root + /bin/mkdir -p /var/tmp + /bin/mkdir /var/tmp/ib-bonding-0.9.0-root + cd ib-bonding-0.9.0 + cd linux/drivers/net/bonding/ ++ pwd + make -C /lib/modules/2.6.16.60-0.14-ppc64/build modules M=/var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding make: Entering directory `/usr/src/linux-2.6.16.60-0.14' WARNING: Symbol version dump /usr/src/linux-2.6.16.60-0.14/Module.symvers is missing; modules will have no dependencies and modversions. CC [M] /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.o /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c:2442:36: error: macro "ip_route_output_key" requires 3 arguments, but only 2 given /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c: In function ‘bond_arp_send_all’: /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c:2442: warning: assignment makes integer from pointer without a cast /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c:3697:44: error: macro "dev_get_by_name" requires 2 arguments, but only 1 given /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c: In function ‘bond_do_ioctl’: /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c:3697: warning: assignment from incompatible pointer type /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c:4266:73: error: macro "INIT_WORK" passed 3 arguments, but takes just 2 /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c: In function ‘bond_init’: /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c:4266: error: ‘INIT_WORK’ undeclared (first use in this function) /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c:4266: error: (Each undeclared identifier is reported only once /var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.c:4266: error: for each function it appears in.) make[1]: *** [/var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding/bond_main.o] Error 1 make: *** [_module_/var/tmp/OFED_topdir/BUILD/ib-bonding-0.9.0/linux/drivers/net/bonding] Error 2 make: Leaving directory `/usr/src/linux-2.6.16.60-0.14' + echo ' Building IB bonding driver failed' Building IB bonding driver failed + exit 1 error: Bad exit status from /var/tmp/rpm-tmp.4571 (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.4571 (%build) ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED-1.4.1 daily: build error in ib-bonding
On Sun, 08 Mar 2009 11:44:49 +0200 Moni Shoua wrote: > > WARNING: Symbol version dump /usr/src/linux-2.6.16.60-0.14/Module.symvers > >is missing; modules will have no dependencies and modversions. > > It seems to me that you are running a beta (rcN) of SLES10 SP2. > Your kernel is 2.6.16.60-0.14-ppc64 while it should be 2.6.16.60-0.21-ppc64 > You can see this in file /etc/issue thanks for the hint, I've updated the install and it works fine now. Regards, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] OFED-1.4.1 build error in openmpi on RHEL5.3
Hi, I am trying to compile the OFED daily build (OFED-1.4.1-20090312-0600) on RHEL 5.3, where I hit the following build error in openmpi: gcc -DHAVE_CONFIG_H -I. -I.. -I../tools/opari/lib -I../extlib/otf/otflib -I../extlib/otf/otflib -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mminimal-toc -m64 -g -O2 -D_GNU_SOURCE -DBINDIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/bin\" -DDATADIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/share\" -DRFG -DVT_MEMHOOK -DVT_IOWRAP -m64 -O2 -MT rfg_filter.o -MD -MP -MF .deps/rfg_filter.Tpo -c -o rfg_filter.o rfg_filter.c vt_iowrap.c:1242: error: expected declaration specifiers or '...' before numeric constant vt_iowrap.c:1243: error: conflicting types for '__fprintf_chk' make[5]: *** [vt_iowrap.o] Error 1 make[5]: *** Waiting for unfinished jobs mv -f .deps/vt_comp_gnu.Tpo .deps/vt_comp_gnu.Po mv -f .deps/rfg_regions.Tpo .deps/rfg_regions.Po mv -f .deps/rfg_filter.Tpo .deps/rfg_filter.Po make[5]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt/vtlib' make[4]: *** [all-recursive] Error 1 make[4]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' make[3]: *** [all] Error 2 make[3]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi' make: *** [all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.76004 (%build) This has already been reported here: http://www.open-mpi.org/community/lists/devel/2009/01/5230.php According to the mail, removing the -D_FORTIFY_SOURCE=2 flag from the spec file works around the issue, will this be done for the next RCs? I appreciate your comments. Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Build problem with rnfs-utils
Hi, i am trying to install OFED-1.4.1-20090325-0600 on a powerpc system with RHEL5.3, including the rnfs-utils package. The build process fails with the following message: checking for powerpc-redhat-linux-gnu-ld... /usr/bin/ld checking for ANSI C header files... (cached) yes checking for GNU libc2... yes checking for BSD signal semantics... no checking for gethostbyname... yes checking for connect... yes checking for getaddrinfo... yes checking for getrpcbynumber... yes checking for getservbyname... yes checking for crypt in -lcrypt... yes checking for blkid_get_library_version in -lblkid... no configure: error: libblkid needed error: Bad exit status from /var/tmp/rpm-tmp.5516 (%build) I have e2fsprogs-devel-1.39-20.el5.ppc64.rpm installed on the system. When I install e2fsprogs-devel-1.39-20.el5.ppc.rpm in addition, the build process works fine. Can this be fixed in the rnfs-utils package so that the 64bit version of e2fsprogs-devel is used or do we need an additional dependency to the 32bit version in the OFED install script? Appreciate your comments, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: Build problem with rnfs-utils
On Thu, 26 Mar 2009 15:09:59 -0500 Jon Mason wrote: > On Thu, Mar 26, 2009 at 10:16:33AM +0100, Alexander Schmidt wrote: > > Hi, > > > > i am trying to install OFED-1.4.1-20090325-0600 on a powerpc system with > > RHEL5.3, including the rnfs-utils package. The build process fails with the > > following message: > > > > checking for powerpc-redhat-linux-gnu-ld... /usr/bin/ld > > checking for ANSI C header files... (cached) yes > > checking for GNU libc2... yes > > checking for BSD signal semantics... no > > checking for gethostbyname... yes > > checking for connect... yes > > checking for getaddrinfo... yes > > checking for getrpcbynumber... yes > > checking for getservbyname... yes > > checking for crypt in -lcrypt... yes > > checking for blkid_get_library_version in -lblkid... no > > configure: error: libblkid needed > > error: Bad exit status from /var/tmp/rpm-tmp.5516 (%build) > > > > I have e2fsprogs-devel-1.39-20.el5.ppc64.rpm installed on the system. When > > I install e2fsprogs-devel-1.39-20.el5.ppc.rpm in addition, the build process > > works fine. Can this be fixed in the rnfs-utils package so that the 64bit > > version of e2fsprogs-devel is used or do we need an additional dependency to > > the 32bit version in the OFED install script? > > It is breaking because the install.pl script is modifying the CFLAGS, > etc for ppc64. I have a patch available which allows the CFLAGS, etc to > be 64bit. This change enables it to compile. I do not know the reasons > why the SRPMS are compiled 32bit on ppc64, so I do not know if this is > the proper fix (or if we need to have the 32bit version of > e2fsprogs-devel). > > Let me know which fix is preferred, and I'll make the necessary change > and push it to Vlad. Well, I also don't know why the CFLAGS are set in the installer or if the 64bit version of rnfs-utils is supposed to work, so I'm not the right person to make that decision. However, as adding a dependency on the e2fsprogs-devel 32bit version seems to be the less intrusive change to me, I would go for that. Thanks, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH] ofed_1_4_scripts: Fix ppc64 building of rnfs-utils
On Fri, 27 Mar 2009 14:52:27 -0500
Jon Mason wrote:
> rnfs-utils will not build on ppc64 without a 32bit version of
> e2fsprogs-devel. The 32bit version is needed due to logic previously
> added to install.pl to force the compile to be 32bit. Another way to
> fix this issue would be to allow 64bit version of the rnfs-utils by
> removing the logic which forces it to be 32bit.
>
> With the added dependency, rnfs-utils compiles on RHEL5.2 on ppc64
> without issue.
>
> Thanks,
> Jon
Hi Jon,
thanks for the patch, it also works fine on RHEL5.3 now.
I've also tried to install the rnfs-utils package on SLES11, and there I run
into another problem. The SRPM has the following dependencies:
[alexs...@bl3d1974 SRPMS]$ rpm -qpR rnfs-utils-1.1.5-rc1.OFED.src.rpm
krb5-devel >= 1.3.1
autoconf >= 2.57
automake
krb5-devel
libtool
krb5-libs
rpmlib(CompressedFileNames) <= 3.0.4-1
I've created the patch below for teaching the installer that rnfs-utils needs
the krb5 packages. But on SLES10 & 11, there is no krb5-libs package, so it's
not possible to install rnfs-utils. Could you try to fix this?
--- OFED-1.4.1-20090329-0600.orig/install.pl
+++ OFED-1.4.1-20090329-0600/install.pl
@@ -947,7 +947,8 @@ my %packages_info = (
{ name => "rnfs-utils", parent => "rnfs-utils",
selected => 0, installed => 0, rpm_exist => 0, rpm_exist32 => 0,
available => 0, mode => "user", dist_req_build => [],
-dist_req_inst => [], ofa_req_build => [],
+dist_req_inst => ( $build32 == 1 )?["krb5-devel$suffix_32bit",
"krb5-libs"]:
+ ["krb5-devel$suffix_64bit", "krb5-libs"], ofa_req_build => [],
ofa_req_inst => [],
install32 => 0, exception => 0, configure_options => '' },
'rnfs-utils-debuginfo' =>
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [Fwd: Re: [ewg] Moving to new server]
On Sun, 29 Mar 2009 11:06:00 -0700 Jeff Becker wrote: > Hi Yannick. Please try again. On Friday, I realized that the home > directory ownership was wrong, and fixed these. Thanks. > Hi Jeff, I'm trying to move the libehca.git tree and I'm also not able to login, on the old server everything works fine... Could you take a look at our account? [alexs...@bl3d1974 ~]$ ssh hngu...@69.55.231.68 hngu...@69.55.231.68's password: Thanks, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH] ofed_1_4_scripts: Fix ppc64 building of rnfs-utils
On Mon, 30 Mar 2009 17:35:58 -0500 Jon Mason wrote: > On Mon, Mar 30, 2009 at 11:25:53AM +0200, Alexander Schmidt wrote: > > On Fri, 27 Mar 2009 14:52:27 -0500 > > Jon Mason wrote: > > > > > rnfs-utils will not build on ppc64 without a 32bit version of > > > e2fsprogs-devel. The 32bit version is needed due to logic previously > > > added to install.pl to force the compile to be 32bit. Another way to > > > fix this issue would be to allow 64bit version of the rnfs-utils by > > > removing the logic which forces it to be 32bit. > > > > > > With the added dependency, rnfs-utils compiles on RHEL5.2 on ppc64 > > > without issue. > > > > > > Thanks, > > > Jon > > > > Hi Jon, > > > > thanks for the patch, it also works fine on RHEL5.3 now. > > > > I've also tried to install the rnfs-utils package on SLES11, and there I run > > into another problem. The SRPM has the following dependencies: > > Thanks for testing this on non-RHEL installs. > > > > > [alexs...@bl3d1974 SRPMS]$ rpm -qpR rnfs-utils-1.1.5-rc1.OFED.src.rpm > > krb5-devel >= 1.3.1 > > autoconf >= 2.57 > > automake > > krb5-devel > > libtool > > krb5-libs > > rpmlib(CompressedFileNames) <= 3.0.4-1 > > > > I've created the patch below for teaching the installer that rnfs-utils > > needs > > the krb5 packages. But on SLES10 & 11, there is no krb5-libs package, so > > it's > > not possible to install rnfs-utils. Could you try to fix this? > > I think the name of the package on SuSE is krb5 or krb5-server. I've > attached a SRPM to this e-mail without the krb5-libs dependency. See if > the package fixes any configure issues you could/will hit when trying to > compile rnfs-utils. You should be able to copy the SRPM over the one in > the OFED tree and it will build as it would normally. > Hi Jon, after installing the tcpd-devel (for libwrap) and krb5 packages I was able to build an RPM from the SRPM you attached. This RPM has a dependency on nfs-utils, which does not exist on SLES11... Seems like they have split the package up?! Could you take care of this so that we get a clean install on SLES11? Thanks, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Moving to new server
On Sun, 29 Mar 2009 21:38:54 -0500 Steve Wise wrote: > Jeff Becker wrote: > > Hi Sasha > > > > Sasha Khapyorsky wrote: > >> Hi Jeff, > >> > >> On 16:24 Fri 27 Mar , Jeff Becker wrote: > >> > >>> Just thought this would be a good chance to clean up. > >>> > >> > >> Thanks for your work. > >> > >> I copied my trees and basically it works. Few things are missing: > >> > >> 1. ~username symbolic links under /pub/scm. So things like: > >> > >> git clone git://69.55.231.68/~sashak/management > >> > >> , don't work yet. > >> > >> 2. Mail server (guess postfix) is not configured and mailx is not > >> installed - both can be useful (and was used on current server) for > >> git commit notification. > >> > >> 3. No gitweb yet. > >> > > > > I plan on setting these up after a good number of developers have > > moved their git trees - just a handful right now. Thanks. > > > Can you setup the /pub/scm/~username links now so we can use the new > server now that we've moved our git trees over? I would also like to get one of these links so that I can test a remote checkout of the tree, the libehca git has now been moved to /home/alexschm/scm/libehca.git In addition, could someone set my default shell to /bin/bash? /bin/sh is a bit limited and I don't want to execute bash everytime I log in ;) Thanks, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH] ofed_1_4_scripts: Fix ppc64 building of rnfs-utils
On Wed, 1 Apr 2009 11:02:03 -0500 Jon Mason wrote: > On Wed, Apr 01, 2009 at 10:39:30AM +0200, Alexander Schmidt wrote: > > On Mon, 30 Mar 2009 17:35:58 -0500 > > Jon Mason wrote: > > > > > On Mon, Mar 30, 2009 at 11:25:53AM +0200, Alexander Schmidt wrote: > > > > On Fri, 27 Mar 2009 14:52:27 -0500 > > > > Jon Mason wrote: > > > > > > > > > rnfs-utils will not build on ppc64 without a 32bit version of > > > > > e2fsprogs-devel. The 32bit version is needed due to logic previously > > > > > added to install.pl to force the compile to be 32bit. Another way to > > > > > fix this issue would be to allow 64bit version of the rnfs-utils by > > > > > removing the logic which forces it to be 32bit. > > > > > > > > > > With the added dependency, rnfs-utils compiles on RHEL5.2 on ppc64 > > > > > without issue. > > > > > > > > > > Thanks, > > > > > Jon > > > > > > > > Hi Jon, > > > > > > > > thanks for the patch, it also works fine on RHEL5.3 now. > > > > > > > > I've also tried to install the rnfs-utils package on SLES11, and there > > > > I run > > > > into another problem. The SRPM has the following dependencies: > > > > > > Thanks for testing this on non-RHEL installs. > > > > > > > > > > > [alexs...@bl3d1974 SRPMS]$ rpm -qpR rnfs-utils-1.1.5-rc1.OFED.src.rpm > > > > krb5-devel >= 1.3.1 > > > > autoconf >= 2.57 > > > > automake > > > > krb5-devel > > > > libtool > > > > krb5-libs > > > > rpmlib(CompressedFileNames) <= 3.0.4-1 > > > > > > > > I've created the patch below for teaching the installer that rnfs-utils > > > > needs > > > > the krb5 packages. But on SLES10 & 11, there is no krb5-libs package, > > > > so it's > > > > not possible to install rnfs-utils. Could you try to fix this? > > > > > > I think the name of the package on SuSE is krb5 or krb5-server. I've > > > attached a SRPM to this e-mail without the krb5-libs dependency. See if > > > the package fixes any configure issues you could/will hit when trying to > > > compile rnfs-utils. You should be able to copy the SRPM over the one in > > > the OFED tree and it will build as it would normally. > > > > > > > Hi Jon, > > > > after installing the tcpd-devel (for libwrap) and krb5 packages I was able > > to > > build an RPM from the SRPM you attached. This RPM has a dependency on > > nfs-utils, > > which does not exist on SLES11... Seems like they have split the package > > up?! > > > > Could you take care of this so that we get a clean install on SLES11? > > I removed the distro specific dependencies from the SRPM and added them > to the install.pl for OFED. I will submit the relevant patch to > install.pl shortly. > > You can get the SRPM of rnfs-utils without the nfs-utils dependency at: > http://www.openfabrics.org/~swise/ofed_1_4/rnfs-utils/ > > Thanks for the testing and let me know if you hit anymore problems. On SLES11 with the rnfs-utils-1.1.5-1.OFED.src.rpm rpm I now get Installation finished successfully. Thanks, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] compile failure on SLES11 in NFS
Hi everybody, with the latest daily build (OFED-1.4.1-20090402-0600) I'm getting the compile problem below. I'm running SLES11 on a powerpc box. Could someone take a look at this? Appreciate your help! Alex -Iarch/powerpc -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/fs/nfs -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -msoft-float -pipe -I/usr/src/linux-2.6.27.19-5/arch/powerpc -Iarch/powerpc -mminimal-toc -mtraceback=none -mcall-aixdesc -mcpu=power4 -mtune=cell -mno-altivec -mno-spe -funit-at-a-time -mno-string -Wa,-maltivec -fno-stack-protector -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(file)" -D"KBUILD_MODNAME=KBUILD_STR(nfs)" -D"DEBUG_HASH=3" -D"DEBUG_HASH2=61" -c -o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/fs/nfs/.tmp_file.o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/fs/nfs/file.c /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/fs/nfs/file.c: In function ‘nfs_write_begin’: /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/fs/nfs/file.c:354: error: implicit declaration of function ‘__grab_cache_page’ /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/fs/nfs/file.c:354: warning: assignment makes pointer from integer without a cast make[5]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/fs/nfs/file.o] Error 1 make[4]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1/fs/nfs] Error 2 make[3]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.4.1] Error 2 make[2]: *** [sub-make] Error 2 make[1]: *** [all] Error 2 make[1]: Leaving directory `/usr/src/linux-2.6.27.19-5-obj/ppc64/ppc64' make: *** [kernel] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.24575 (%build) ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [Fwd: [ewg] compile failure on SLES11 in NFS]
On Fri, 03 Apr 2009 09:06:11 -0700 Jeff Becker wrote: > Hi Jon > > Jon Mason wrote: > > Hey Jeff, > > I just wanted to make you aware of the issue below. I thought I broke > > something until I saw the "SLES11". Did you already do the SLES11 > > backport, or are they trying to build something that is not completed? > > > > I did see this. The problem is that I don't have a powerpc to test > builds on. The x86-64 SLES11 backport is done. Can someone with access > to powerpc help out here? Thanks. I had a look at this and noticed that the x86-64 backport patch fixes the issue I saw, it just needs to get applied ;-) Vlad, how about this? I do not know the exact kernel version of the SLES11 release, therefore I've inserted some asterisks. Signed-off-by: Alexander Schmidt --- ofed_scripts/get_backport_dir.sh |3 +++ 1 file changed, 3 insertions(+) --- ofa_kernel-1.4.1.orig/ofed_scripts/get_backport_dir.sh +++ ofa_kernel-1.4.1/ofed_scripts/get_backport_dir.sh @@ -117,6 +117,9 @@ get_backport_dir() 2.6.26*) echo 2.6.26 ;; +2.6.27.*-*) +echo 2.6.27_sles11 +;; *) echo ;; ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH] ofed_1_4_scripts: Fix ppc64 building of rnfs-utils for SLES
Fix typo in package dependency description.
Signed-off-by: Alexander Schmidt
---
install.pl |4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--- OFED-1.4.1-20090406-0600.orig/install.pl
+++ OFED-1.4.1-20090406-0600/install.pl
@@ -2651,8 +2651,8 @@ sub check_linux_dependencies
print RED "krb5 is required to build rnfs-utils.",
RESET "\n";
$err++;
}
-if ($arch eq "ppc64" and not is_installed("tcp-devel")) {
-print RED "tcp-devel is required to build
rnfs-utils.", RESET "\n";
+if ($arch eq "ppc64" and not is_installed("tcpd-devel")) {
+print RED "tcpd-devel is required to build
rnfs-utils.", RESET "\n";
$err++;
}
}
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
Hi Roland,
did you have a chance to take a look at the patchset and will you apply it, or
are there any outstanding issues we need to address?
Regards,
Alex
On Wed, 22 Apr 2009 16:02:28 +0200
Stefan Roscher wrote:
> In case of large queue pairs there is the possibillity of allocation failures
> due to memory fragmentationo with kmalloc().To ensure the memory is allocated
> even
> if kmalloc() can not find chunks which are big enough, we try to allocate the
> memory
> with vmalloc().
>
> Signed-off-by: Stefan Roscher
> ---
>
> On Tuesday 21 April 2009 07:34:30 pm Roland Dreier wrote:
> > > +queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *),
> > GFP_KERNEL);
> >
> > How big might this buffer be? Any chance of allocation failure due to
> > memory fragmentation?
> >
> > - R.
> Hey Roland,
> yes you are right and here is the patch to circumvent the described problem.
> It will apply on top of the patchset.
> regards Stefan
>
>
>
> drivers/infiniband/hw/ehca/ipz_pt_fn.c | 17 +
> 1 files changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> index a260559..1227c59 100644
> --- a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> +++ b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> @@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
> *queue,
> /* allocate queue page pointers */
> queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
> if (!queue->queue_pages) {
> - ehca_gen_err("Couldn't allocate queue page list");
> - return 0;
> + queue->queue_pages = vmalloc(nr_of_pages * sizeof(void *));
> + if (!queue->queue_pages) {
> + ehca_gen_err("Couldn't allocate queue page list");
> + return 0;
> + }
> }
> memset(queue->queue_pages, 0, nr_of_pages * sizeof(void *));
>
> @@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
> *queue,
> ipz_queue_ctor_exit0:
> ehca_gen_err("Couldn't alloc pages queue=%p "
>"nr_of_pages=%x", queue, nr_of_pages);
> - kfree(queue->queue_pages);
> + if (is_vmalloc_addr(queue->queue_pages))
> + vfree(queue->queue_pages);
> + else
> + kfree(queue->queue_pages);
>
> return 0;
> }
> @@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue
> *queue)
> free_page((unsigned long)queue->queue_pages[i]);
> }
>
> - kfree(queue->queue_pages);
> + if (is_vmalloc_addr(queue->queue_pages))
> + vfree(queue->queue_pages);
> + else
> + kfree(queue->queue_pages);
>
> return 1;
> }
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
On Tue, 28 Apr 2009 07:01:32 -0700 Roland Dreier wrote: > > did you have a chance to take a look at the patchset and will you apply > it, or > > are there any outstanding issues we need to address? > > I guess it's OK, but definitely 2.6.31 material. I guess I'll stick it > linux-next soon. > > - R. Okay with us, thank you very much! Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Please update RN for OFED 1.4.1
On Sun, 10 May 2009 17:23:29 +0300 Tziporet Koren wrote: > Hi > > I still miss update of some RN toward teh coming GA release > > * uDAPL - Arlin > * Open MPI 1.3.1 (and update other MPI docs to reflect the change of > MPI version) - Jeff S. > * ipath driver - Qlogic (Ralph?) > * ehca driver - IBM (Hoang-Nam) > * RDS - Andy > * Bonding - Moni Hi Tziporet, thank you for the reminder, here are the changes for ehca. [PATCH] ehca: release notes for OFED-1.4.1 Signed-off-by: Alexander Schmidt --- ehca_release_notes.txt | 17 - 1 files changed, 12 insertions(+), 5 deletions(-) diff --git a/ehca_release_notes.txt b/ehca_release_notes.txt index 263fec6..17025b4 100644 --- a/ehca_release_notes.txt +++ b/ehca_release_notes.txt @@ -1,8 +1,8 @@ Open Fabrics Enterprise Distribution (OFED) -ehca in OFED 1.4 Release Notes +ehca in OFED 1.4.1 Release Notes - December 2008 + May 2009 Overview @@ -38,7 +38,11 @@ whereby is one of the following items: New Features -- Flush error completion support +- none + +Fixed Bugs ofed-1.4.1 +- +- none Fixed Bugs ofed-1.4 - @@ -77,8 +81,9 @@ Fixed Bugs ofed-1.3 Available backports --- -- RedHat EL5 up1: 2.6.18-53.ELsmp - RedHat EL5 up2: 2.6.18-92.ELsmp +- RedHat EL5 up3: 2.6.18-128.ELsmp +- SLES11: 2.6.27.19-5.1-smp - SLES10SP1: 2.6.16-53-0.16-smp - SLES10SP2: 2.6.16-60 - kernel.org: 2.6.24-27 @@ -93,4 +98,6 @@ strongly recommended to set option nr_ports=-1 to enable autodetect mode: loading the ehca device driver. 3. Dynamic memory operations are not supported with ehca - + +4. Allocating a large number of queue pairs might be time consuming. This will +be fixed in next OFED release. -- 1.5.5 ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED libraries - on the web at last
On Wed, 10 Jun 2009 14:20:24 +0300 "Tziporet Koren" wrote: > Hi All > I have placed the list of OFED public and private libraries on the web: > http://www.openfabrics.org/txt/documentation/linux/ofed_libs.txt > > Please review and update me if there is any change to this list > > Tziporet > Hi Tziporet, could you explain what "public" and "private" means and how the libraries are classified? If this has something to do with the new model for pulling the libraries into OFED, it would be especially interesting to get more information, but this might be another topic... Thanks, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH 2.6.31] ehca: Tolerate dynamic memory operations and huge pages
Hi Roland,
thank you for taking a look at the code!
On Fri, 12 Jun 2009 21:50:58 -0700
Roland Dreier wrote:
> OK, one major issue with this patch and a few minor nits.
>
> First, the major issue is that I don't see anything in the patch that
> changes the code in ehca_mem_notifier() in ehca_main.c:
>
> case MEM_GOING_ONLINE:
> case MEM_GOING_OFFLINE:
> /* only ok if no hca is attached to the lpar */
> spin_lock_irqsave(&shca_list_lock, flags);
> if (list_empty(&shca_list)) {
> spin_unlock_irqrestore(&shca_list_lock, flags);
> return NOTIFY_OK;
> } else {
> spin_unlock_irqrestore(&shca_list_lock, flags);
> if (printk_timed_ratelimit(&ehca_dmem_warn_time,
> 30 * 1000))
> ehca_gen_err("DMEM operations are not allowed"
>"in conjunction with eHCA");
> return NOTIFY_BAD;
> }
>
> But your patch description says:
>
> > This patch implements toleration of dynamic memory operations
>
> But it seems you're still going to hit the same NOTIFY_BAD case above
> after your patch. So something doesn't compute for me. Could you
> explain more?
Yeah, the notifier code remains untouched as we still do not allow dynamic
memory operations _while_ our module is loaded. The patch allows the driver to
cope with DMEM operations that happened before the module was loaded, which
might result in a non-contiguous memory layout. When the driver registers
its global memory region in the system, the memory layout must be considered.
We chose the term "toleration" instead of "support" to illustrate this.
I'll put some more details into the changelog, incorporate the other comments
and send out a second version of the patch.
Thanks,
Alex
>
> Second, a nit:
>
> > +#define EHCA_REG_MR 0
> > +#define EHCA_REG_BUSMAP_MR (~0)
>
> and you pass these as the reg_busmap parm in:
>
> > int ehca_reg_mr(struct ehca_shca *shca,
> >struct ehca_mr *e_mr,
> >u64 *iova_start,
> > @@ -991,7 +1031,8 @@
> >struct ehca_pd *e_pd,
> >struct ehca_mr_pginfo *pginfo,
> >u32 *lkey, /*OUT*/
> > - u32 *rkey) /*OUT*/
> > + u32 *rkey, /*OUT*/
> > + int reg_busmap)
>
> and test it as:
>
> > + if (reg_busmap)
> > + ret = ehca_reg_bmap_mr_rpages(shca, e_mr, pginfo);
> > + else
> > + ret = ehca_reg_mr_rpages(shca, e_mr, pginfo);
>
> So the ~0 for true looks a bit odd. One option would be to make
> reg_busmap a bool, since that's how you're using it, but then you lose
> the nice self-documenting macro where you call things.
>
> So I think it would be cleaner to do something like
>
> enum ehca_reg_type {
> EHCA_REG_MR,
> EHCA_REG_BUSMAP_MR
> };
>
> and make the "int reg_busmap" parameter into "enum ehca_reg_type reg_type"
> and have the code become
>
> + if (reg_type == EHCA_REG_BUSMAP_MR)
> + ret = ehca_reg_bmap_mr_rpages(shca, e_mr, pginfo);
> + else if (reg_type == EHCA_REG_MR)
> + ret = ehca_reg_mr_rpages(shca, e_mr, pginfo);
> + else
> + ret = -EINVAL
>
> or something like that.
>
> > +struct ib_dma_mapping_ops ehca_dma_mapping_ops = {
> > + .mapping_error = ehca_dma_mapping_error,
> > + .map_single = ehca_dma_map_single,
> > + .unmap_single = ehca_dma_unmap_single,
> > + .map_page = ehca_dma_map_page,
> > + .unmap_page = ehca_dma_unmap_page,
> > + .map_sg = ehca_dma_map_sg,
> > + .unmap_sg = ehca_dma_unmap_sg,
> > + .dma_address = ehca_dma_address,
> > + .dma_len = ehca_dma_len,
> > + .sync_single_for_cpu = ehca_dma_sync_single_for_cpu,
> > + .sync_single_for_device = ehca_dma_sync_single_for_device,
> > + .alloc_coherent = ehca_dma_alloc_coherent,
> > + .free_coherent = ehca_dma_free_coherent,
> > +};
>
> I always think structures like this are easier to read if you align the
> '=' signs. But no big deal.
>
> > + ret = ehca_create_busmap();
> > + if (ret) {
> > + ehca_gen_err("Cannot create busmap.");
> > + goto module_init2;
> > + }
> > +
> >ret = ibmebus_register_driver(&ehca_driver);
> >if (ret) {
> >ehca_gen_err("Cannot register eHCA device driver");
> >ret = -EINVAL;
> > - goto module_init2;
> > + goto module_init3;
> >}
> >
> >ret = register_memory_notifier(&ehca_mem_nb);
> >if (ret) {
> >ehca_gen_err("Failed registering memory add/remove notifier");
> > - goto module_init3;
> > + goto module_init4;
>
> Having to renumber unrelated things is when something changes is why I
> don't like this style of error path labels. But I think it's well and
> truly too late to fix that in e
[ewg] Re: [PATCH 2.6.31 try 2] ehca: Tolerate dynamic memory operations and huge pages
From: Hannes Hering
This patch implements toleration of dynamic memory operations and 16 GB
gigantic pages. "Toleration" means that the driver can cope with dynamic
memory operations that happened before the driver was loaded. While using the
ehca driver, dynamic memory operations are still prohibited. On module load the
driver walks through available system memory, checks for available memory ranges
and then registers the kernel internal memory region accordingly. The
translation of address ranges is implemented via a 3-level busmap.
Signed-off-by: Hannes Hering
---
This patch is built and tested against infiniband.git. Please apply for 2.6.31.
drivers/infiniband/hw/ehca/ehca_main.c | 20 +
drivers/infiniband/hw/ehca/ehca_mrmw.c | 508 -
drivers/infiniband/hw/ehca/ehca_mrmw.h | 13
3 files changed, 523 insertions(+), 18 deletions(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_main.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_main.c
@@ -52,7 +52,7 @@
#include "ehca_tools.h"
#include "hcp_if.h"
-#define HCAD_VERSION "0026"
+#define HCAD_VERSION "0027"
MODULE_LICENSE("Dual BSD/GPL");
MODULE_AUTHOR("Christoph Raisch ");
@@ -506,6 +506,7 @@ static int ehca_init_device(struct ehca_
shca->ib_device.detach_mcast= ehca_detach_mcast;
shca->ib_device.process_mad = ehca_process_mad;
shca->ib_device.mmap= ehca_mmap;
+ shca->ib_device.dma_ops = &ehca_dma_mapping_ops;
if (EHCA_BMASK_GET(HCA_CAP_SRQ, shca->hca_cap)) {
shca->ib_device.uverbs_cmd_mask |=
@@ -1028,17 +1029,23 @@ static int __init ehca_module_init(void)
goto module_init1;
}
+ ret = ehca_create_busmap();
+ if (ret) {
+ ehca_gen_err("Cannot create busmap.");
+ goto module_init2;
+ }
+
ret = ibmebus_register_driver(&ehca_driver);
if (ret) {
ehca_gen_err("Cannot register eHCA device driver");
ret = -EINVAL;
- goto module_init2;
+ goto module_init3;
}
ret = register_memory_notifier(&ehca_mem_nb);
if (ret) {
ehca_gen_err("Failed registering memory add/remove notifier");
- goto module_init3;
+ goto module_init4;
}
if (ehca_poll_all_eqs != 1) {
@@ -1053,9 +1060,12 @@ static int __init ehca_module_init(void)
return 0;
-module_init3:
+module_init4:
ibmebus_unregister_driver(&ehca_driver);
+module_init3:
+ ehca_destroy_busmap();
+
module_init2:
ehca_destroy_slab_caches();
@@ -1073,6 +1083,8 @@ static void __exit ehca_module_exit(void
unregister_memory_notifier(&ehca_mem_nb);
+ ehca_destroy_busmap();
+
ehca_destroy_slab_caches();
ehca_destroy_comp_pool();
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_mrmw.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_mrmw.c
@@ -53,6 +53,38 @@
/* max number of rpages (per hcall register_rpages) */
#define MAX_RPAGES 512
+/* DMEM toleration management */
+#define EHCA_SECTSHIFTSECTION_SIZE_BITS
+#define EHCA_SECTSIZE (1UL << EHCA_SECTSHIFT)
+#define EHCA_HUGEPAGESHIFT 34
+#define EHCA_HUGEPAGE_SIZE (1UL << EHCA_HUGEPAGESHIFT)
+#define EHCA_HUGEPAGE_PFN_MASK ((EHCA_HUGEPAGE_SIZE - 1) >> PAGE_SHIFT)
+#define EHCA_INVAL_ADDR0xULL
+#define EHCA_DIR_INDEX_SHIFT 13 /* 8k Entries in 64k block */
+#define EHCA_TOP_INDEX_SHIFT (EHCA_DIR_INDEX_SHIFT * 2)
+#define EHCA_MAP_ENTRIES (1 << EHCA_DIR_INDEX_SHIFT)
+#define EHCA_TOP_MAP_SIZE (0x1) /* currently fixed map size
*/
+#define EHCA_DIR_MAP_SIZE (0x1)
+#define EHCA_ENT_MAP_SIZE (0x1)
+#define EHCA_INDEX_MASK (EHCA_MAP_ENTRIES - 1)
+
+static unsigned long ehca_mr_len;
+
+/*
+ * Memory map data structures
+ */
+struct ehca_dir_bmap {
+ u64 ent[EHCA_MAP_ENTRIES];
+};
+struct ehca_top_bmap {
+ struct ehca_dir_bmap *dir[EHCA_MAP_ENTRIES];
+};
+struct ehca_bmap {
+ struct ehca_top_bmap *top[EHCA_MAP_ENTRIES];
+};
+
+static struct ehca_bmap *ehca_bmap;
+
static struct kmem_cache *mr_cache;
static struct kmem_cache *mw_cache;
@@ -68,6 +100,8 @@ enum ehca_mr_pgsize {
#define EHCA_MR_PGSHIFT1M 20
#define EHCA_MR_PGSHIFT16M 24
+static u64 ehca_map_vaddr(void *caddr);
+
static u32 ehca_encode_hwpage_size(u32 pgsize)
{
int log = ilog2(pgsize);
@@ -135,7 +169,8 @@ struct ib_mr *ehca_get_dma_mr(struct ib_
goto get_dma_mr_exit0;
}
- ret = ehca_reg_maxmr(shca, e_maxmr, (u64 *)KERNELBASE,
+ ret = ehca_reg_maxmr(shca, e_maxmr,
+(void *)ehca_map_vaddr((void *)KERNELBASE),
mr_access_flags, e_pd,
&e_maxmr->ib.ib_mr.lk
Re: [ewg] Re: [PATCH 2.6.31] ehca: Tolerate dynamic memory operations and huge pages
On Tue, 16 Jun 2009 09:10:39 -0700 Roland Dreier wrote: > > > Yeah, the notifier code remains untouched as we still do not allow dynamic > > memory operations _while_ our module is loaded. The patch allows the > driver to > > cope with DMEM operations that happened before the module was loaded, which > > might result in a non-contiguous memory layout. When the driver registers > > its global memory region in the system, the memory layout must be > considered. > > > > We chose the term "toleration" instead of "support" to illustrate this. > > I see. So things just silently broke in some cases when the driver was > loaded after operations you didn't tolerate? > > Anyway, thanks for the explanation. Well, things did not break silently. The registration of the MR failed with an error code which was reported to userspace. Will you push the patch for .31 or .32? Thanks, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [PATCH 2.6.31 try 2] ehca: Tolerate dynamic memory operations and huge pages
On Mon, 22 Jun 2009 22:19:21 -0700
Roland Dreier wrote:
> thanks, applied.
>
> > -#define HCAD_VERSION "0026"
> > +#define HCAD_VERSION "0027"
>
> the driver version was already 0027 (since bde2cfaf), so I dropped this chunk.
thank you for applying, we would like to increase the version number for this
patch, so please also apply the following:
ehca: Increment version number for DMEM toleration
Signed-off-by: Alexander Schmidt
---
drivers/infiniband/hw/ehca/ehca_main.c |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_main.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_main.c
@@ -52,7 +52,7 @@
#include "ehca_tools.h"
#include "hcp_if.h"
-#define HCAD_VERSION "0027"
+#define HCAD_VERSION "0028"
MODULE_LICENSE("Dual BSD/GPL");
MODULE_AUTHOR("Christoph Raisch ");
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Distribution of userspace code for OFED-1.5
Hi all, some time ago a new model for providing code updates in userspace libraries was decided upon for OFED-1.5. In this model, the distributors of userspace libraries should upload release tarballs to directories on the ofed server and new code should be picked up from there. I haven't heard any news on this topic since OFED-1.5 development has been started and we have new code for the ehca library pending, so my question is: is this new process already established or should we continue with sending out git pull requests? I think the distributors were in favor of the new model, so maybe they have an opinion on this? I appreciate any comments! Regards, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] ehca: use port autodetect mode as default
This patch sets the port autodetect mode as default for the ehca driver. The
autodetect code has been in the kernel for several releases now and has
proved to be stable.
---
Roland, please queue this change for 2.6.32 if you are okay with it.
drivers/infiniband/hw/ehca/ehca_main.c |8
1 file changed, 4 insertions(+), 4 deletions(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_main.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_main.c
@@ -52,7 +52,7 @@
#include "ehca_tools.h"
#include "hcp_if.h"
-#define HCAD_VERSION "0028"
+#define HCAD_VERSION "0029"
MODULE_LICENSE("Dual BSD/GPL");
MODULE_AUTHOR("Christoph Raisch ");
@@ -64,7 +64,7 @@ static int ehca_hw_level = 0;
static int ehca_poll_all_eqs = 1;
int ehca_debug_level = 0;
-int ehca_nr_ports = 2;
+int ehca_nr_ports = -1;
int ehca_use_hp_mr = 0;
int ehca_port_act_time = 30;
int ehca_static_rate = -1;
@@ -95,8 +95,8 @@ MODULE_PARM_DESC(hw_level,
"Hardware level (0: autosensing (default), "
"0x10..0x14: eHCA, 0x20..0x23: eHCA2)");
MODULE_PARM_DESC(nr_ports,
-"number of connected ports (-1: autodetect, 1: port one only, "
-"2: two ports (default)");
+"number of connected ports (-1: autodetect (default), "
+"1: port one only, 2: two ports)");
MODULE_PARM_DESC(use_hp_mr,
"Use high performance MRs (default: no)");
MODULE_PARM_DESC(port_act_time,
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] OFED-1.5: build problem with mthca
Hi all, with the latest OFED-1.5 daily build on SLES11 (powerpc64), I get the following build problem. Could someone have a look at this? Thanks, Alex -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/include \ -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/debug \ -I/usr/local/include/scst \ -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/ulp/srpt \ -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/net/cxgb3 \ -Iinclude \ -Iinclude2 -I/usr/src/linux-2.6.27.19-5/include \ -I/usr/src/linux-2.6.27.19-5/arch/powerpc/include \ -Iarch/powerpc -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os -msoft-float -pipe -I/usr/src/linux-2.6.27.19-5/arch/powerpc -Iarch/powerpc -mminimal-toc -mtraceback=none -mcall-aixdesc -mcpu=power4 -mtune=cell -mno-altivec -mno-spe -funit-at-a-time -mno-string -Wa,-maltivec -fno-stack-protector -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(mthca_eq)" -D"KBUILD_MODNAME=KBUILD_STR(ib_mthca)" -D"DEBUG_HASH=48" -D"DEBUG_HASH2=63" -c -o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/.tmp_mthca_eq.o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c In file included from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h:6, from /usr/src/linux-2.6.27.19-5/include/linux/interrupt.h:9, from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c:35: include2/asm/topology.h:75: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘dump_numa_cpu_topology’ make[6]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.o] Error 1 make[5]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca] Error 2 make[4]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband] Error 2 make[3]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5] Error 2 make[2]: *** [sub-make] Error 2 make[1]: *** [all] Error 2 make[1]: Leaving directory `/usr/src/linux-2.6.27.19-5-obj/ppc64/ppc64' make: *** [kernel] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.68731 (%build) ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH OFED-1.5] ehca: sync with mainline
Hi Vlad,
please apply the following patch for OFED-1.5 to get the latest fixes for the
ehca driver.
Thanks,
Alex
Index: ofa_kernel-1.5/kernel_patches/fixes/ehca-0010-replace_vmalloc.patch
===
--- /dev/null 1970-01-01 00:00:00.0 +
+++ ofa_kernel-1.5/kernel_patches/fixes/ehca-0010-replace_vmalloc.patch
2009-07-24 04:39:54.0 -0400
@@ -0,0 +1,43 @@
+commit bf31a1a02eb28d9bda0bb74345df7889faeb7335
+Author: Anton Blanchard
+Date: Wed May 13 16:52:40 2009 -0700
+
+IB/ehca: Replace vmalloc() with kmalloc() for queue allocation
+
+To improve performance of driver resource allocation, replace
+vmalloc() calls with kmalloc().
+
+Signed-off-by: Stefan Roscher
+Signed-off-by: Roland Dreier
+
+diff --git a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
+index c3a3284..a260559 100644
+--- a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
+@@ -220,7 +220,7 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
*queue,
+ queue->small_page = NULL;
+
+ /* allocate queue page pointers */
+- queue->queue_pages = vmalloc(nr_of_pages * sizeof(void *));
++ queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
+ if (!queue->queue_pages) {
+ ehca_gen_err("Couldn't allocate queue page list");
+ return 0;
+@@ -240,7 +240,7 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
*queue,
+ ipz_queue_ctor_exit0:
+ ehca_gen_err("Couldn't alloc pages queue=%p "
+"nr_of_pages=%x", queue, nr_of_pages);
+- vfree(queue->queue_pages);
++ kfree(queue->queue_pages);
+
+ return 0;
+ }
+@@ -262,7 +262,7 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue
*queue)
+ free_page((unsigned long)queue->queue_pages[i]);
+ }
+
+- vfree(queue->queue_pages);
++ kfree(queue->queue_pages);
+
+ return 1;
+ }
Index:
ofa_kernel-1.5/kernel_patches/fixes/ehca-0020-vmalloc_for_big_allocation.patch
===
--- /dev/null 1970-01-01 00:00:00.0 +
+++
ofa_kernel-1.5/kernel_patches/fixes/ehca-0020-vmalloc_for_big_allocation.patch
2009-07-24 04:39:54.0 -0400
@@ -0,0 +1,56 @@
+commit c94f156f63c835ffc02b686f9d4238b106f31a5d
+Author: Stefan Roscher
+Date: Wed May 13 16:52:42 2009 -0700
+
+IB/ehca: Fall back to vmalloc() for big allocations
+
+In case of large queue pairs there is the possibillity of allocation
+failures due to memory fragmentation when using kmalloc(). To ensure
+the memory is allocated even if kmalloc() can not find chunks which
+are big enough, we fall back to allocating the memory with vmalloc().
+
+Signed-off-by: Stefan Roscher
+Signed-off-by: Roland Dreier
+
+diff --git a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
+index a260559..1227c59 100644
+--- a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
+@@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
*queue,
+ /* allocate queue page pointers */
+ queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
+ if (!queue->queue_pages) {
+- ehca_gen_err("Couldn't allocate queue page list");
+- return 0;
++ queue->queue_pages = vmalloc(nr_of_pages * sizeof(void *));
++ if (!queue->queue_pages) {
++ ehca_gen_err("Couldn't allocate queue page list");
++ return 0;
++ }
+ }
+ memset(queue->queue_pages, 0, nr_of_pages * sizeof(void *));
+
+@@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
*queue,
+ ipz_queue_ctor_exit0:
+ ehca_gen_err("Couldn't alloc pages queue=%p "
+"nr_of_pages=%x", queue, nr_of_pages);
+- kfree(queue->queue_pages);
++ if (is_vmalloc_addr(queue->queue_pages))
++ vfree(queue->queue_pages);
++ else
++ kfree(queue->queue_pages);
+
+ return 0;
+ }
+@@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue
*queue)
+ free_page((unsigned long)queue->queue_pages[i]);
+ }
+
+- kfree(queue->queue_pages);
++ if (is_vmalloc_addr(queue->queue_pages))
++ vfree(queue->queue_pages);
++ else
++ kfree(queue->queue_pages);
+
+ return 1;
+ }
Index:
ofa_kernel-1.5/kernel_patches/fixes/ehca-0030-remove_allocation_for_user_qp.patch
===
--- /dev/null 1970-01-01 00:00:00.0 +
+++
ofa_kernel-1.5/kernel_patches/fixes/ehca-0030-remove_allocation_for_user_qp.patch
2009-07-24 04:39:54.0 -0400
@@ -0,0 +
Re: [ewg] OFED-1.5: build problem with mthca
Hi Jack, On Wed, 22 Jul 2009 10:58:36 +0300 Jack Morgenstein wrote: > Your compiler seems to be choking on the "__init" keyword in the kernel > installation's > asm/topology.h file. I do not know why. The kernel_addons/.../cpumask.h > file simply > has a line #include , which should not be causing problems. > > Problem line: > extern void __init dump_numa_cpu_topology(void); I'm using the stock SLES11 gcc compiler, and still see this in OFED-1.5-20090722-0600. Could you take another look? Regards, Alex > > -Jack > > On Monday 13 July 2009 18:48, Alexander Schmidt wrote: > > Hi all, > > > > with the latest OFED-1.5 daily build on SLES11 (powerpc64), I get the > > following build problem. Could someone have a look at this? > > > > Thanks, > > Alex > > > > > > -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/include \ > > -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/debug \ > > -I/usr/local/include/scst \ > > -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/ulp/srpt \ > > -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/net/cxgb3 \ > > -Iinclude \ > > -Iinclude2 -I/usr/src/linux-2.6.27.19-5/include \ > > -I/usr/src/linux-2.6.27.19-5/arch/powerpc/include \ > > -Iarch/powerpc > > -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca > > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing > > -fno-common -Werror-implicit-function-declaration -Os -msoft-float -pipe > > -I/usr/src/linux-2.6.27.19-5/arch/powerpc -Iarch/powerpc -mminimal-toc > > -mtraceback=none -mcall-aixdesc -mcpu=power4 -mtune=cell -mno-altivec > > -mno-spe -funit-at-a-time -mno-string -Wa,-maltivec -fno-stack-protector > > -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign > > -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(mthca_eq)" > > -D"KBUILD_MODNAME=KBUILD_STR(ib_mthca)" -D"DEBUG_HASH=48" > > -D"DEBUG_HASH2=63" -c -o > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/.tmp_mthca_eq.o > > > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c > > In file included from > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h:6, > > from > > /usr/src/linux-2.6.27.19-5/include/linux/interrupt.h:9, > > from > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c:35: > > include2/asm/topology.h:75: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or > > ‘__attribute__’ before ‘dump_numa_cpu_topology’ > > make[6]: *** > > [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.o] > > Error 1 > > make[5]: *** > > [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca] > > Error 2 > > make[4]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband] > > Error 2 > > make[3]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5] Error 2 > > make[2]: *** [sub-make] Error 2 > > make[1]: *** [all] Error 2 > > make[1]: Leaving directory `/usr/src/linux-2.6.27.19-5-obj/ppc64/ppc64' > > make: *** [kernel] Error 2 > > error: Bad exit status from /var/tmp/rpm-tmp.68731 (%build) > > ___ > > ewg mailing list > > ewg@lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > > ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH OFED-1.5] eHCA: SLES11 backports
Hi Vlad,
please apply the following patch to get our SLES11 backports.
Thanks,
Alex
Index:
ofa_kernel-1.5.patch2/kernel_patches/backport/2.6.27_sles11/ehca-020-undo_unsigned_long.patch
===
--- /dev/null 1970-01-01 00:00:00.0 +
+++
ofa_kernel-1.5.patch2/kernel_patches/backport/2.6.27_sles11/ehca-020-undo_unsigned_long.patch
2009-07-27 09:12:20.0 -0400
@@ -0,0 +1,1005 @@
+Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_cq.c
+===
+--- ofa_kernel-1.5.orig/drivers/infiniband/hw/ehca/ehca_cq.c 2009-07-26
09:08:48.0 -0400
ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_cq.c2009-07-27
08:59:04.0 -0400
+@@ -196,7 +196,7 @@
+
+ if (h_ret != H_SUCCESS) {
+ ehca_err(device, "hipz_h_alloc_resource_cq() failed "
+- "h_ret=%lli device=%p", h_ret, device);
++ "h_ret=%li device=%p", h_ret, device);
+ cq = ERR_PTR(ehca2ib_return_code(h_ret));
+ goto create_cq_exit2;
+ }
+@@ -232,7 +232,7 @@
+
+ if (h_ret < H_SUCCESS) {
+ ehca_err(device, "hipz_h_register_rpage_cq() failed "
+- "ehca_cq=%p cq_num=%x h_ret=%lli counter=%i "
++ "ehca_cq=%p cq_num=%x h_ret=%li counter=%i "
+"act_pages=%i", my_cq, my_cq->cq_number,
+h_ret, counter, param.act_pages);
+ cq = ERR_PTR(-EINVAL);
+@@ -244,7 +244,7 @@
+ if ((h_ret != H_SUCCESS) || vpage) {
+ ehca_err(device, "Registration of pages not "
+"complete ehca_cq=%p cq_num=%x "
+- "h_ret=%lli", my_cq, my_cq->cq_number,
++ "h_ret=%li", my_cq, my_cq->cq_number,
+h_ret);
+ cq = ERR_PTR(-EAGAIN);
+ goto create_cq_exit4;
+@@ -252,7 +252,7 @@
+ } else {
+ if (h_ret != H_PAGE_REGISTERED) {
+ ehca_err(device, "Registration of page failed "
+- "ehca_cq=%p cq_num=%x h_ret=%lli "
++ "ehca_cq=%p cq_num=%x h_ret=%li "
+"counter=%i act_pages=%i",
+my_cq, my_cq->cq_number,
+h_ret, counter, param.act_pages);
+@@ -266,7 +266,7 @@
+
+ gal = my_cq->galpas.kernel;
+ cqx_fec = hipz_galpa_load(gal, CQTEMM_OFFSET(cqx_fec));
+- ehca_dbg(device, "ehca_cq=%p cq_num=%x CQX_FEC=%llx",
++ ehca_dbg(device, "ehca_cq=%p cq_num=%x CQX_FEC=%lx",
+my_cq, my_cq->cq_number, cqx_fec);
+
+ my_cq->ib_cq.cqe = my_cq->nr_of_entries =
+@@ -307,7 +307,7 @@
+ h_ret = hipz_h_destroy_cq(adapter_handle, my_cq, 1);
+ if (h_ret != H_SUCCESS)
+ ehca_err(device, "hipz_h_destroy_cq() failed ehca_cq=%p "
+- "cq_num=%x h_ret=%lli", my_cq, my_cq->cq_number,
h_ret);
++ "cq_num=%x h_ret=%li", my_cq, my_cq->cq_number, h_ret);
+
+ create_cq_exit2:
+ write_lock_irqsave(&ehca_cq_idr_lock, flags);
+@@ -355,7 +355,7 @@
+ h_ret = hipz_h_destroy_cq(adapter_handle, my_cq, 0);
+ if (h_ret == H_R_STATE) {
+ /* cq in err: read err data and destroy it forcibly */
+- ehca_dbg(device, "ehca_cq=%p cq_num=%x resource=%llx in err "
++ ehca_dbg(device, "ehca_cq=%p cq_num=%x ressource=%lx in err "
+"state. Try to delete it forcibly.",
+my_cq, cq_num, my_cq->ipz_cq_handle.handle);
+ ehca_error_data(shca, my_cq, my_cq->ipz_cq_handle.handle);
+@@ -365,7 +365,7 @@
+cq_num);
+ }
+ if (h_ret != H_SUCCESS) {
+- ehca_err(device, "hipz_h_destroy_cq() failed h_ret=%lli "
++ ehca_err(device, "hipz_h_destroy_cq() failed h_ret=%li "
+"ehca_cq=%p cq_num=%x", h_ret, my_cq, cq_num);
+ return ehca2ib_return_code(h_ret);
+ }
+Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_hca.c
+===
+--- ofa_kernel-1.5.orig/drivers/infiniband/hw/ehca/ehca_hca.c 2009-07-26
09:08:48.0 -0400
ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_hca.c 2009-07-27
08:59:04.0 -0400
+@@ -393,7 +393,7 @@
+ hret = hipz_h_modify_port(shca->ipz_hca_handle, port,
+ cap, props->init_type, port_modify_mask);
+ if (hret != H_SUCCESS) {
+-
[ewg] Please pull libehca.git
Hi Vlad, please pull the libehca.git to get the following fix for OFED-1.5. git://git.openfabrics.org/~hnguyen/libehca.git master Alexander Schmidt (1): libehca: SRQ overflow prevention src/ehca_qes.h |1 + src/ehca_uclasses.h |8 src/ehca_umain.c| 41 - src/ehca_ureqs.c| 34 ++ 4 files changed, 79 insertions(+), 5 deletions(-) ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] New libehca tarball
Hi Vlad, please include the new libehca tarball for OFED-1.5: http://www.openfabrics.org/downloads/libehca/libehca-1.2.1-0.1.g0a82a52.tar.gz It contains the following changes: Joachim Fenkes (1): libehca: Fix CQE flags reporting Kind regards, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED 1.5 beta status
On Wed, 9 Sep 2009 16:47:14 +0300 "Tziporet Koren" wrote: Hi, > Hi, > I wish to update all that we plan to release OFED 1.5 beta tomorrow > > I know it's a week late then what we planned but we waited that all > modules will at least pass compilation on all supported OSes before we > publish the beta the mthca driver does not compile yet on SLES11 on powerpc. I've already reported this twice. -I/usr/src/linux-2.6.27.23-0.1/arch/powerpc/include \ -Iarch/powerpc -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -fwrapv -Os -msoft-float -pipe -I/usr/src/linux-2.6.27.23-0.1/arch/powerpc -Iarch/powerpc -mminimal-toc -mtraceback=none -mcall-aixdesc -mcpu=power4 -mtune=cell -mno-altivec -mno-spe -funit-at-a-time -mno-string -Wa,-maltivec -fno-stack-protector -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -fwrapv -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(mthca_eq)" -D"KBUILD_MODNAME=KBUILD_STR(ib_mthca)" -D"DEBUG_HASH=48" -D"DEBUG_HASH2=63" -c -o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/.tmp_mthca_eq.o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c In file included from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h:6, from /usr/src/linux-2.6.27.23-0.1/include/linux/interrupt.h:9, from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c:35: include2/asm/topology.h:75: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘dump_numa_cpu_topology’ make[6]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.o] Error 1 make[5]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca] Error 2 make[4]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband] Error 2 make[3]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5] Error 2 make[2]: *** [sub-make] Error 2 make[1]: *** [all] Error 2 make[1]: Leaving directory `/usr/src/linux-2.6.27.23-0.1-obj/ppc64/ppc64' make: *** [kernel] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.19962 (%build) ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED 1.5 beta status
On Thu, 10 Sep 2009 11:41:10 +0300 Tziporet Koren wrote: > Alexander Schmidt wrote: > > On Wed, 9 Sep 2009 16:47:14 +0300 > > "Tziporet Koren" wrote: > > Hi, > > > > > >> Hi, > >> I wish to update all that we plan to release OFED 1.5 beta tomorrow > >> > >> I know it's a week late then what we planned but we waited that all > >> modules will at least pass compilation on all supported OSes before we > >> publish the beta > >> > > > > the mthca driver does not compile yet on SLES11 on powerpc. I've already > > reported this twice. > > > > -I/usr/src/linux-2.6.27.23-0.1/arch/powerpc/include \ > > -Iarch/powerpc > > -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca > > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing > > -fno-common -Werror-implicit-function-declaration -fwrapv -Os -msoft-float > > -pipe -I/usr/src/linux-2.6.27.23-0.1/arch/powerpc -Iarch/powerpc > > -mminimal-toc -mtraceback=none -mcall-aixdesc -mcpu=power4 -mtune=cell > > -mno-altivec -mno-spe -funit-at-a-time -mno-string -Wa,-maltivec > > -fno-stack-protector -fomit-frame-pointer -g -Wdeclaration-after-statement > > -Wno-pointer-sign -fwrapv -DMODULE -D"KBUILD_STR(s)=#s" > > -D"KBUILD_BASENAME=KBUILD_STR(mthca_eq)" > > -D"KBUILD_MODNAME=KBUILD_STR(ib_mthca)" -D"DEBUG_HASH=48" > > -D"DEBUG_HASH2=63" -c -o > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/.tmp_mthca_eq.o > > > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c > > In file included from > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h:6, > > from > > /usr/src/linux-2.6.27.23-0.1/include/linux/interrupt.h:9, > > from > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c:35: > > include2/asm/topology.h:75: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or > > ‘__attribute__’ before ‘dump_numa_cpu_topology’ > > make[6]: *** > > [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.o] > > Error 1 > > make[5]: *** > > [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca] > > Error 2 > > make[4]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband] > > Error 2 > > make[3]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5] Error 2 > > make[2]: *** [sub-make] Error 2 > > make[1]: *** [all] Error 2 > > make[1]: Leaving directory `/usr/src/linux-2.6.27.23-0.1-obj/ppc64/ppc64' > > make: *** [kernel] Error 2 > > error: Bad exit status from /var/tmp/rpm-tmp.19962 (%build) > > > > > > Sorry we missed this > I will put this limitation in the release mail and Jack will fix it soon > > Tziporet The following change fixes the issue for me, and it did not break other parts of the stack, could someone review this? Thanks Index: ofa_kernel-1.5/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h === --- ofa_kernel-1.5.orig/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h +++ ofa_kernel-1.5/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h @@ -3,7 +3,6 @@ #include_next #include -#include #define cpumask_of(cpu)(get_cpu_mask(cpu)) #define cpumask_of_node(node) (_node_to_cpumask_ptr(node)) ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED 1.5 beta status
On Thu, 10 Sep 2009 18:18:46 -0500 Jon Mason wrote: > On Thu, Sep 10, 2009 at 11:18:43AM -0500, Jon Mason wrote: > > On Thu, Sep 10, 2009 at 06:11:48PM +0200, Alexander Schmidt wrote: > > > On Thu, 10 Sep 2009 11:41:10 +0300 > > > Tziporet Koren wrote: > > > > > > > Alexander Schmidt wrote: > > > > > On Wed, 9 Sep 2009 16:47:14 +0300 > > > > > "Tziporet Koren" wrote: > > > > > Hi, > > > > > > > > > > > > > > >> Hi, > > > > >> I wish to update all that we plan to release OFED 1.5 beta tomorrow > > > > >> > > > > >> I know it's a week late then what we planned but we waited that all > > > > >> modules will at least pass compilation on all supported OSes before > > > > >> we > > > > >> publish the beta > > > > >> > > > > > > > > > > the mthca driver does not compile yet on SLES11 on powerpc. I've > > > > > already > > > > > reported this twice. > > > > > > > > > > -I/usr/src/linux-2.6.27.23-0.1/arch/powerpc/include \ > > > > > -Iarch/powerpc > > > > > -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca > > > > > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs > > > > > -fno-strict-aliasing -fno-common > > > > > -Werror-implicit-function-declaration -fwrapv -Os -msoft-float -pipe > > > > > -I/usr/src/linux-2.6.27.23-0.1/arch/powerpc -Iarch/powerpc > > > > > -mminimal-toc -mtraceback=none -mcall-aixdesc -mcpu=power4 > > > > > -mtune=cell -mno-altivec -mno-spe -funit-at-a-time -mno-string > > > > > -Wa,-maltivec -fno-stack-protector -fomit-frame-pointer -g > > > > > -Wdeclaration-after-statement -Wno-pointer-sign -fwrapv -DMODULE > > > > > -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(mthca_eq)" > > > > > -D"KBUILD_MODNAME=KBUILD_STR(ib_mthca)" -D"DEBUG_HASH=48" > > > > > -D"DEBUG_HASH2=63" -c -o > > > > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/.tmp_mthca_eq.o > > > > > > > > > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c > > > > > In file included from > > > > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h:6, > > > > > from > > > > > /usr/src/linux-2.6.27.23-0.1/include/linux/interrupt.h:9, > > > > > from > > > > > /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.c:35: > > > > > include2/asm/topology.h:75: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or > > > > > ‘__attribute__’ before ‘dump_numa_cpu_topology’ > > > > > make[6]: *** > > > > > [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca/mthca_eq.o] > > > > > Error 1 > > > > > make[5]: *** > > > > > [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband/hw/mthca] > > > > > Error 2 > > > > > make[4]: *** > > > > > [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5/drivers/infiniband] Error 2 > > > > > make[3]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5] > > > > > Error 2 > > > > > make[2]: *** [sub-make] Error 2 > > > > > make[1]: *** [all] Error 2 > > > > > make[1]: Leaving directory > > > > > `/usr/src/linux-2.6.27.23-0.1-obj/ppc64/ppc64' > > > > > make: *** [kernel] Error 2 > > > > > error: Bad exit status from /var/tmp/rpm-tmp.19962 (%build) > > > > > > > > > > > > > > > > > > Sorry we missed this > > > > I will put this limitation in the release mail and Jack will fix it soon > > > > > > > > Tziporet > > > > > > The following change fixes the issue for me, and it did not break other > > > parts of > > > the stack, could someone review this? > > > > > > Thanks > > > > > > Index: > > > ofa_kernel-1.5/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h > > > === > > > --- > > > ofa_kernel-1.5.orig/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h > > > +++ > > > ofa_kernel-1.5/kernel_addons/backport/2.6.27_sles11/include/linux/cpumask.h > > > @@ -3,7 +3,6 @@ > > > > > > #include_next > > > #include > > > -#include > > > > > > #define cpumask_of(cpu) (get_cpu_mask(cpu)) > > > #define cpumask_of_node(node) (_node_to_cpumask_ptr(node)) > > > > I believe this will break compilation on x86_64, but I will verify. > > It seems to compile fine, go ahead and commit it. > > Thanks, > Jon Thanks for verifying this. Vlad, if you are okay with it, please apply. Regards, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] ehca: fixes for OFED-1.5
Hi Vlad,
please apply the following fixes for OFED-1.5, thanks!
Index: ofa_kernel-1.5/kernel_patches/fixes/ehca-0080-mad_redirection.patch
===
--- /dev/null
+++ ofa_kernel-1.5/kernel_patches/fixes/ehca-0080-mad_redirection.patch
@@ -0,0 +1,102 @@
+commit d706834d995939c96b4952bf042918879b4db18b
+Author: Joachim Fenkes
+Date: Sat Sep 5 20:22:55 2009 -0700
+
+IB/ehca: Construct MAD redirect replies from request MAD
+
+The old code used a lot of hard-coded values, which might not be valid
+in all environments (especially routed fabrics or partitioned
+subnets). Copy as much information as possible from the incoming
+request to correct that.
+
+Signed-off-by: Joachim Fenkes
+Signed-off-by: Roland Dreier
+
+diff --git a/drivers/infiniband/hw/ehca/ehca_sqp.c
b/drivers/infiniband/hw/ehca/ehca_sqp.c
+index c568b28..8c1213f 100644
+--- a/drivers/infiniband/hw/ehca/ehca_sqp.c
b/drivers/infiniband/hw/ehca/ehca_sqp.c
+@@ -125,14 +125,30 @@ struct ib_perf {
+ u8 data[192];
+ } __attribute__ ((packed));
+
++/* TC/SL/FL packed into 32 bits, as in ClassPortInfo */
++struct tcslfl {
++ u32 tc:8;
++ u32 sl:4;
++ u32 fl:20;
++} __attribute__ ((packed));
++
++/* IP Version/TC/FL packed into 32 bits, as in GRH */
++struct vertcfl {
++ u32 ver:4;
++ u32 tc:8;
++ u32 fl:20;
++} __attribute__ ((packed));
+
+ static int ehca_process_perf(struct ib_device *ibdev, u8 port_num,
++ struct ib_wc *in_wc, struct ib_grh *in_grh,
+struct ib_mad *in_mad, struct ib_mad *out_mad)
+ {
+ struct ib_perf *in_perf = (struct ib_perf *)in_mad;
+ struct ib_perf *out_perf = (struct ib_perf *)out_mad;
+ struct ib_class_port_info *poi =
+ (struct ib_class_port_info *)out_perf->data;
++ struct tcslfl *tcslfl =
++ (struct tcslfl *)&poi->redirect_tcslfl;
+ struct ehca_shca *shca =
+ container_of(ibdev, struct ehca_shca, ib_device);
+ struct ehca_sport *sport = &shca->sport[port_num - 1];
+@@ -158,10 +174,29 @@ static int ehca_process_perf(struct ib_device *ibdev, u8
port_num,
+ poi->base_version = 1;
+ poi->class_version = 1;
+ poi->resp_time_value = 18;
+- poi->redirect_lid = sport->saved_attr.lid;
+- poi->redirect_qp = sport->pma_qp_nr;
++
++ /* copy local routing information from WC where applicable */
++ tcslfl->sl = in_wc->sl;
++ poi->redirect_lid =
++ sport->saved_attr.lid | in_wc->dlid_path_bits;
++ poi->redirect_qp = sport->pma_qp_nr;
+ poi->redirect_qkey = IB_QP1_QKEY;
+- poi->redirect_pkey = IB_DEFAULT_PKEY_FULL;
++
++ ehca_query_pkey(ibdev, port_num, in_wc->pkey_index,
++ &poi->redirect_pkey);
++
++ /* if request was globally routed, copy route info */
++ if (in_grh) {
++ struct vertcfl *vertcfl =
++ (struct vertcfl *)&in_grh->version_tclass_flow;
++ memcpy(poi->redirect_gid, in_grh->dgid.raw,
++ sizeof(poi->redirect_gid));
++ tcslfl->tc= vertcfl->tc;
++ tcslfl->fl= vertcfl->fl;
++ } else
++ /* else only fill in default GID */
++ ehca_query_gid(ibdev, port_num, 0,
++ (union ib_gid *)&poi->redirect_gid);
+
+ ehca_dbg(ibdev, "ehca_pma_lid=%x ehca_pma_qp=%x",
+sport->saved_attr.lid, sport->pma_qp_nr);
+@@ -183,8 +218,7 @@ perf_reply:
+
+ int ehca_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
+struct ib_wc *in_wc, struct ib_grh *in_grh,
+- struct ib_mad *in_mad,
+- struct ib_mad *out_mad)
++ struct ib_mad *in_mad, struct ib_mad *out_mad)
+ {
+ int ret;
+
+@@ -196,7 +230,8 @@ int ehca_process_mad(struct ib_device *ibdev, int
mad_flags, u8 port_num,
+ return IB_MAD_RESULT_SUCCESS;
+
+ ehca_dbg(ibdev, "port_num=%x src_qp=%x", port_num, in_wc->src_qp);
+- ret = ehca_process_perf(ibdev, port_num, in_mad, out_mad);
++ ret = ehca_process_perf(ibdev, port_num, in_wc, in_grh,
++ in_mad, out_mad);
+
+ return ret;
+ }
Index: ofa_kernel-1.5/kernel_patches/fixes/ehca-0090-cqe_flag_reporting.patch
===
--- /dev/null
+++ ofa_kernel-1.5/kernel_patches/fixes/ehca-0090-cqe_flag_reporting.patch
@@ -0,0 +1,29 @@
+commit 6303e74c699d6ba8bd3d44ec6898b2d6aa55d788
+Author: Joachim Fenkes
+Date: Sat Sep 5 20:22:55 2009 -0700
+
+IB/ehca: F
[ewg] [PATCH] libehca supported kernel versions
Hi Vlad,
please apply the following patch for install.pl.
Signed-off-by: Alexander Schmidt
Index: OFED-1.5-20090915-0844/install.pl
===
--- OFED-1.5-20090915-0844.orig/install.pl
+++ OFED-1.5-20090915-0844/install.pl
@@ -1646,10 +1646,8 @@ sub set_availability
set_compilers();
# Ehca
-# if ($arch =~ m/ppc64|powerpc/ and
-# $kernel =~ m/2.6.1[6-9]|2.6.2[0-9]/) {
if ($arch =~ m/ppc64|powerpc/ and
-$kernel =~ m/2.6.30/) {
+$kernel =~ m/2.6.1[6-9]|2.6.2[0-9]|2.6.30/) {
$kernel_modules_info{'ehca'}{'available'} = 1;
$packages_info{'libehca'}{'available'} = 1;
$packages_info{'libehca-devel-static'}{'available'} = 1;
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] ehca: backports for 2.6.27
Hi Vlad,
please apply the following ehca backports for 2.6.27. Thanks!
Signed-off-by: Alexander Schmidt
Index: ofa_kernel-1.5/kernel_patches/backport/2.6.27/ehca-010-undo_cpumask.patch
===
--- /dev/null
+++ ofa_kernel-1.5/kernel_patches/backport/2.6.27/ehca-010-undo_cpumask.patch
@@ -0,0 +1,42 @@
+---
+ drivers/infiniband/hw/ehca/ehca_irq.c | 14 --
+ 1 file changed, 8 insertions(+), 6 deletions(-)
+
+Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_irq.c
+===
+--- ofa_kernel-1.5.orig/drivers/infiniband/hw/ehca/ehca_irq.c 2009-07-27
08:20:08.0 -0400
ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_irq.c 2009-07-27
08:26:31.0 -0400
+@@ -659,12 +659,12 @@
+
+ WARN_ON_ONCE(!in_interrupt());
+ if (ehca_debug_level >= 3)
+- ehca_dmp(cpu_online_mask, cpumask_size(), "");
++ ehca_dmp(&cpu_online_map, sizeof(cpumask_t), "");
+
+ spin_lock_irqsave(&pool->last_cpu_lock, flags);
+- cpu = cpumask_next(pool->last_cpu, cpu_online_mask);
++ cpu = next_cpu_nr(pool->last_cpu, cpu_online_map);
+ if (cpu >= nr_cpu_ids)
+- cpu = cpumask_first(cpu_online_mask);
++ cpu = first_cpu(cpu_online_map);
+ pool->last_cpu = cpu;
+ spin_unlock_irqrestore(&pool->last_cpu_lock, flags);
+
+@@ -855,7 +855,7 @@
+ case CPU_UP_CANCELED_FROZEN:
+ ehca_gen_dbg("CPU: %x (CPU_CANCELED)", cpu);
+ cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
+- kthread_bind(cct->task, cpumask_any(cpu_online_mask));
++ kthread_bind(cct->task, any_online_cpu(cpu_online_map));
+ destroy_comp_task(pool, cpu);
+ break;
+ case CPU_ONLINE:
+@@ -902,7 +902,7 @@
+ return -ENOMEM;
+
+ spin_lock_init(&pool->last_cpu_lock);
+- pool->last_cpu = cpumask_any(cpu_online_mask);
++ pool->last_cpu = any_online_cpu(cpu_online_map);
+
+ pool->cpu_comp_tasks = alloc_percpu(struct ehca_cpu_comp_task);
+ if (pool->cpu_comp_tasks == NULL) {
Index:
ofa_kernel-1.5/kernel_patches/backport/2.6.27/ehca-020-undo_unsigned_long.patch
===
--- /dev/null
+++
ofa_kernel-1.5/kernel_patches/backport/2.6.27/ehca-020-undo_unsigned_long.patch
@@ -0,0 +1,1005 @@
+Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_cq.c
+===
+--- ofa_kernel-1.5.orig/drivers/infiniband/hw/ehca/ehca_cq.c 2009-07-26
09:08:48.0 -0400
ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_cq.c2009-07-27
08:59:04.0 -0400
+@@ -196,7 +196,7 @@
+
+ if (h_ret != H_SUCCESS) {
+ ehca_err(device, "hipz_h_alloc_resource_cq() failed "
+- "h_ret=%lli device=%p", h_ret, device);
++ "h_ret=%li device=%p", h_ret, device);
+ cq = ERR_PTR(ehca2ib_return_code(h_ret));
+ goto create_cq_exit2;
+ }
+@@ -232,7 +232,7 @@
+
+ if (h_ret < H_SUCCESS) {
+ ehca_err(device, "hipz_h_register_rpage_cq() failed "
+- "ehca_cq=%p cq_num=%x h_ret=%lli counter=%i "
++ "ehca_cq=%p cq_num=%x h_ret=%li counter=%i "
+"act_pages=%i", my_cq, my_cq->cq_number,
+h_ret, counter, param.act_pages);
+ cq = ERR_PTR(-EINVAL);
+@@ -244,7 +244,7 @@
+ if ((h_ret != H_SUCCESS) || vpage) {
+ ehca_err(device, "Registration of pages not "
+"complete ehca_cq=%p cq_num=%x "
+- "h_ret=%lli", my_cq, my_cq->cq_number,
++ "h_ret=%li", my_cq, my_cq->cq_number,
+h_ret);
+ cq = ERR_PTR(-EAGAIN);
+ goto create_cq_exit4;
+@@ -252,7 +252,7 @@
+ } else {
+ if (h_ret != H_PAGE_REGISTERED) {
+ ehca_err(device, "Registration of page failed "
+- "ehca_cq=%p cq_num=%x h_ret=%lli "
++ "ehca_cq=%p cq_num=%x h_ret=%li "
+"counter=%i act_pages=%i",
+my_cq, my_cq->cq_number,
+h_ret, counter, pa
Re: [ewg] Agenda for EWG/OFED meeting on next Monday
Hi All, we will run the following tests: OS: - RHEL5: up3, up4 - SLES10 SP2, SP3 - SLES11 - Kernel.org: 2.6.27, 2.6.28, 2.6.29, 2.6.30 Arch: - ppc64 (build base and build all test) HCA: - eHCA ULPs: - IPoIB (netpipe, netperf) ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] ib/ehca: rework destroy_eq()
The ibmebus_free_irq() function, which might sleep, was called with
interrupts disabled. To ship around this, make sure that no interrupts
are running by killing the interrupt tasklet. Also lock the shca_list_lock to
protect against the poll_eqs_timer running concurrently.
Signed-off-by: Alexander Schmidt
---
Roland, please apply this for your "next" tree, thanks.
drivers/infiniband/hw/ehca/ehca_classes.h |1 +
drivers/infiniband/hw/ehca/ehca_eq.c |9 ++---
drivers/infiniband/hw/ehca/ehca_main.c|2 +-
3 files changed, 8 insertions(+), 4 deletions(-)
--- linux-2.6.orig/drivers/infiniband/hw/ehca/ehca_eq.c
+++ linux-2.6/drivers/infiniband/hw/ehca/ehca_eq.c
@@ -169,12 +169,15 @@ int ehca_destroy_eq(struct ehca_shca *sh
unsigned long flags;
u64 h_ret;
- spin_lock_irqsave(&eq->spinlock, flags);
ibmebus_free_irq(eq->ist, (void *)shca);
- h_ret = hipz_h_destroy_eq(shca->ipz_hca_handle, eq);
+ spin_lock_irqsave(&shca_list_lock, flags);
+ eq->is_initialized = 0;
+ spin_unlock_irqrestore(&shca_list_lock, flags);
+
+ tasklet_kill(&eq->interrupt_task);
- spin_unlock_irqrestore(&eq->spinlock, flags);
+ h_ret = hipz_h_destroy_eq(shca->ipz_hca_handle, eq);
if (h_ret != H_SUCCESS) {
ehca_err(&shca->ib_device, "Can't free EQ resources.");
--- linux-2.6.orig/drivers/infiniband/hw/ehca/ehca_classes.h
+++ linux-2.6/drivers/infiniband/hw/ehca/ehca_classes.h
@@ -375,6 +375,7 @@ extern rwlock_t ehca_qp_idr_lock;
extern rwlock_t ehca_cq_idr_lock;
extern struct idr ehca_qp_idr;
extern struct idr ehca_cq_idr;
+extern spinlock_t shca_list_lock;
extern int ehca_static_rate;
extern int ehca_port_act_time;
--- linux-2.6.orig/drivers/infiniband/hw/ehca/ehca_main.c
+++ linux-2.6/drivers/infiniband/hw/ehca/ehca_main.c
@@ -123,7 +123,7 @@ DEFINE_IDR(ehca_qp_idr);
DEFINE_IDR(ehca_cq_idr);
static LIST_HEAD(shca_list); /* list of all registered ehcas */
-static DEFINE_SPINLOCK(shca_list_lock);
+DEFINE_SPINLOCK(shca_list_lock);
static struct timer_list poll_eqs_timer;
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] docs: update ehca release notes for OFED-1.5
Hi Vlad, Signed-off-by: Alexander Schmidt --- ehca_release_notes.txt | 33 + 1 file changed, 17 insertions(+), 16 deletions(-) --- docs.git.orig/ehca_release_notes.txt +++ docs.git/ehca_release_notes.txt @@ -1,8 +1,8 @@ Open Fabrics Enterprise Distribution (OFED) -ehca in OFED 1.4.1 Release Notes +ehca in OFED 1.5 Release Notes - May 2009 + December 2009 Overview @@ -28,7 +28,7 @@ whereby is one of the follow - scaling_codescaling code (0: disable (default), 1: enable) - open_aqp1 Open AQP1 on startup (default: no) (bool) - hw_levelHardware level (0: autosensing (default), 0x10..0x14: eHCA, 0x20..0x23: eHCA2) (int) -- nr_portsnumber of connected ports (-1: autodetect, 1: port one only, 2: two ports (default) (int) +- nr_portsnumber of connected ports (-1: autodetect (default), 1: port one only, 2: two ports) (int) - use_hp_mr Use high performance MRs (default: no) (bool) - poll_all_eqsPoll all event queues periodically (default: yes) (bool) - static_rate Set permanent static rate (default: no static rate) (int) @@ -38,7 +38,14 @@ whereby is one of the follow New Features -- none +- DMEM toleration +- Port autodetection set to default + +Fixed Bugs ofed-1.5 +- +- SRQ overflow prevention +- Performance improvements for QP creation +- MAD redirection fix Fixed Bugs ofed-1.4.1 - @@ -81,23 +88,17 @@ Fixed Bugs ofed-1.3 Available backports --- -- RedHat EL5 up2: 2.6.18-92.ELsmp - RedHat EL5 up3: 2.6.18-128.ELsmp +- RedHat EL5 up4: 2.6.18-164.ELsmp - SLES11: 2.6.27.19-5.1-smp -- SLES10SP1: 2.6.16-53-0.16-smp - SLES10SP2: 2.6.16-60 -- kernel.org: 2.6.24-27 +- SLES10SP3: 2.6.16.60-0.54.5 +- kernel.org: 2.6.27-30 Known Issues -1. The device driver normally uses both ports. For using just one port it is -strongly recommended to set option nr_ports=-1 to enable autodetect mode: - modprobe ib_ehca nr_ports=-1 - -2. Furthermore the port(s) needs to be connected to an active switch port while +1. The port(s) needs to be connected to an active switch port while loading the ehca device driver. -3. Dynamic memory operations are not supported with ehca - -4. Allocating a large number of queue pairs might be time consuming. This will -be fixed in next OFED release. +2. Dynamic memory operations are tolerated by ehca, but are prevented by +the driver while it is loaded. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] ib/ehca: fix in_wc handling in process_mad()
If the caller does not pass a valid in_wc to process_mad(),
return MAD failure as it is not possible to generate a valid
MAD redirect response.
Signed-off-by: Alexander Schmidt
---
Hi Roland,
this is another patch we would like to get in your next tree for
2.6.34.
drivers/infiniband/hw/ehca/ehca_sqp.c |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- linux-2.6.orig/drivers/infiniband/hw/ehca/ehca_sqp.c
+++ linux-2.6/drivers/infiniband/hw/ehca/ehca_sqp.c
@@ -222,7 +222,7 @@ int ehca_process_mad(struct ib_device *i
{
int ret;
- if (!port_num || port_num > ibdev->phys_port_cnt)
+ if (!port_num || port_num > ibdev->phys_port_cnt || !in_wc)
return IB_MAD_RESULT_FAILURE;
/* accept only pma request */
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH OFED-151] ehca fixes
Hi Vlad,
please apply the following fixes for OFED-1.5.1, thank you!
Regards,
Alex
diff -Nurp
ofa_kernel-1.5.1.old/kernel_patches/fixes/ehca-0100-rework_destroy_eq.patch
ofa_kernel-1.5.1/kernel_patches/fixes/ehca-0100-rework_destroy_eq.patch
--- ofa_kernel-1.5.1.old/kernel_patches/fixes/ehca-0100-rework_destroy_eq.patch
1970-01-01 01:00:00.0 +0100
+++ ofa_kernel-1.5.1/kernel_patches/fixes/ehca-0100-rework_destroy_eq.patch
2010-02-15 11:43:55.0 +0100
@@ -0,0 +1,63 @@
+commit 9420269428b3dc80c98e52beac60a3976fbef7d2
+Author: Alexander Schmidt
+Date: Wed Dec 9 10:11:04 2009 -0800
+
+IB/ehca: Rework destroy_eq()
+
+The ibmebus_free_irq() function, which might sleep, was called with
+interrupts disabled. To fix this, make sure that no interrupts are
+running by killing the interrupt tasklet. Also lock the
+shca_list_lock to protect against the poll_eqs_timer running
+concurrently.
+
+Signed-off-by: Alexander Schmidt
+Signed-off-by: Roland Dreier
+
+diff --git a/drivers/infiniband/hw/ehca/ehca_classes.h
b/drivers/infiniband/hw/ehca/ehca_classes.h
+index c825142..0136abd 100644
+--- a/drivers/infiniband/hw/ehca/ehca_classes.h
b/drivers/infiniband/hw/ehca/ehca_classes.h
+@@ -375,6 +375,7 @@ extern rwlock_t ehca_qp_idr_lock;
+ extern rwlock_t ehca_cq_idr_lock;
+ extern struct idr ehca_qp_idr;
+ extern struct idr ehca_cq_idr;
++extern spinlock_t shca_list_lock;
+
+ extern int ehca_static_rate;
+ extern int ehca_port_act_time;
+diff --git a/drivers/infiniband/hw/ehca/ehca_eq.c
b/drivers/infiniband/hw/ehca/ehca_eq.c
+index 523e733..3b87589 100644
+--- a/drivers/infiniband/hw/ehca/ehca_eq.c
b/drivers/infiniband/hw/ehca/ehca_eq.c
+@@ -169,12 +169,15 @@ int ehca_destroy_eq(struct ehca_shca *shca, struct
ehca_eq *eq)
+ unsigned long flags;
+ u64 h_ret;
+
+- spin_lock_irqsave(&eq->spinlock, flags);
+ ibmebus_free_irq(eq->ist, (void *)shca);
+
+- h_ret = hipz_h_destroy_eq(shca->ipz_hca_handle, eq);
++ spin_lock_irqsave(&shca_list_lock, flags);
++ eq->is_initialized = 0;
++ spin_unlock_irqrestore(&shca_list_lock, flags);
+
+- spin_unlock_irqrestore(&eq->spinlock, flags);
++ tasklet_kill(&eq->interrupt_task);
++
++ h_ret = hipz_h_destroy_eq(shca->ipz_hca_handle, eq);
+
+ if (h_ret != H_SUCCESS) {
+ ehca_err(&shca->ib_device, "Can't free EQ resources.");
+diff --git a/drivers/infiniband/hw/ehca/ehca_main.c
b/drivers/infiniband/hw/ehca/ehca_main.c
+index fb2d83c..129a6be 100644
+--- a/drivers/infiniband/hw/ehca/ehca_main.c
b/drivers/infiniband/hw/ehca/ehca_main.c
+@@ -123,7 +123,7 @@ DEFINE_IDR(ehca_qp_idr);
+ DEFINE_IDR(ehca_cq_idr);
+
+ static LIST_HEAD(shca_list); /* list of all registered ehcas */
+-static DEFINE_SPINLOCK(shca_list_lock);
++DEFINE_SPINLOCK(shca_list_lock);
+
+ static struct timer_list poll_eqs_timer;
+
diff -Nurp
ofa_kernel-1.5.1.old/kernel_patches/fixes/ehca-0110-dont_turnoff_irq_in_tasklet.patch
ofa_kernel-1.5.1/kernel_patches/fixes/ehca-0110-dont_turnoff_irq_in_tasklet.patch
---
ofa_kernel-1.5.1.old/kernel_patches/fixes/ehca-0110-dont_turnoff_irq_in_tasklet.patch
1970-01-01 01:00:00.0 +0100
+++
ofa_kernel-1.5.1/kernel_patches/fixes/ehca-0110-dont_turnoff_irq_in_tasklet.patch
2010-02-15 11:43:55.0 +0100
@@ -0,0 +1,33 @@
+rq_spinlock is only taken in tasklet context, so it is safe not to
+disable hardware interrupts.
+
+Signed-off-by: Alexander Schmidt
+---
+ drivers/infiniband/hw/ehca/ehca_irq.c |5 ++---
+ 1 file changed, 2 insertions(+), 3 deletions(-)
+
+--- linux-2.6.orig/drivers/infiniband/hw/ehca/ehca_irq.c
linux-2.6/drivers/infiniband/hw/ehca/ehca_irq.c
+@@ -548,11 +548,10 @@ void ehca_process_eq(struct ehca_shca *s
+ struct ehca_eq *eq = &shca->eq;
+ struct ehca_eqe_cache_entry *eqe_cache = eq->eqe_cache;
+ u64 eqe_value, ret;
+- unsigned long flags;
+ int eqe_cnt, i;
+ int eq_empty = 0;
+
+- spin_lock_irqsave(&eq->irq_spinlock, flags);
++ spin_lock(&eq->irq_spinlock);
+ if (is_irq) {
+ const int max_query_cnt = 100;
+ int query_cnt = 0;
+@@ -643,7 +642,7 @@ void ehca_process_eq(struct ehca_shca *s
+ } while (1);
+
+ unlock_irq_spinlock:
+- spin_unlock_irqrestore(&eq->irq_spinlock, flags);
++ spin_unlock(&eq->irq_spinlock);
+ }
+
+ void ehca_tasklet_eq(unsigned long data)
+
diff -Nurp
ofa_kernel-1.5.1.old/kernel_patches/fixes/ehca-0120-allow_access_by_query_qp.patch
ofa_kernel-1.5.1/kernel_patches/fixes/ehca-0120-allow_access_by_query_qp.patch
---
ofa_kernel-1.5.1.old/kernel_patches/fixes/ehca-0120-allow_access_by_query_qp.patch
1970-01-01 01:00:00.0 +0100
+++
ofa_kernel-1.5.1/kernel_patches/fixes/ehca-0120-allow_access_by_query_qp.patch
2010-02-15 11:43:55.0
Re: [ewg] [PATCH OFED-151] ehca fixes
On Tue, 16 Feb 2010 15:29:11 +0200 Vladimir Sokolovsky wrote: > Alexander Schmidt wrote: > > Hi Vlad, > > > > please apply the following fixes for OFED-1.5.1, thank you! > > > > Regards, > > Alex > > > > Hi Alex, > These fixes require updates in backport patches, at least for SLES10 SP2 and > SP3: > > > /tmp/ofa_1_5_dev_kernel-20100216-1441_linux-2.6.16.60-0.54.5-smp_check/kernel_patches/backport/2.6.16_sles10_sp3/ehca-030-ibmebus_loc_code.patch > /usr/bin/quilt --quiltrc > /tmp/ofa_1_5_dev_kernel-20100216-1441_linux-2.6.16.60-0.54.5-smp_check/patches/quiltrc > import > /tmp/ofa_1_5_dev_kernel-20100216-1441_linux-2.6.16.60-0.54.5-smp_check/kernel_patches/backport/2.6.16_sles10_sp3/ehca-030-ibmebus_loc_code.patch > Importing patch > /tmp/ofa_1_5_dev_kernel-20100216-1441_linux-2.6.16.60-0.54.5-smp_check/kernel_patches/backport/2.6.16_sles10_sp3/ehca-030-ibmebus_loc_code.patch > (stored as ehca-030-ibmebus_loc_code.patch) > /usr/bin/quilt --quiltrc > /tmp/ofa_1_5_dev_kernel-20100216-1441_linux-2.6.16.60-0.54.5-smp_check/patches/quiltrc > push patches/ehca-030-ibmebus_loc_code.patch > Applying patch ehca-030-ibmebus_loc_code.patch > patching file drivers/infiniband/hw/ehca/ehca_classes.h > patching file drivers/infiniband/hw/ehca/ehca_eq.c > Hunk #3 FAILED at 170. > 1 out of 3 hunks FAILED -- rejects in file > drivers/infiniband/hw/ehca/ehca_eq.c > patching file drivers/infiniband/hw/ehca/ehca_main.c > Patch ehca-030-ibmebus_loc_code.patch does not apply (enforce with -f) > > You can reproduce it by: > # ./ofed_scripts/ofed_makedist.sh > > Regards, > Vladimir Hi Vlad, thank you for checking this. I've included an update of the backports in the patch below, ofed_makedist.sh runs fine now. Regards, Alex diff --git a/kernel_patches/backport/2.6.16_sles10_sp2/ehca-030-ibmebus_loc_code.patch b/kernel_patches/backport/2.6.16_sles10_sp2/ehca-030-ibmebus_loc_code.patch index 9d6830d..23a7f1e 100644 --- a/kernel_patches/backport/2.6.16_sles10_sp2/ehca-030-ibmebus_loc_code.patch +++ b/kernel_patches/backport/2.6.16_sles10_sp2/ehca-030-ibmebus_loc_code.patch @@ -39,15 +39,15 @@ Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_eq.c IRQF_DISABLED, "ehca_neq", (void *)shca); if (ret < 0) -@@ -170,7 +170,7 @@ - u64 h_ret; - - spin_lock_irqsave(&eq->spinlock, flags); -- ibmebus_free_irq(eq->ist, (void *)shca); -+ ibmebus_free_irq(NULL, eq->ist, (void *)shca); - - h_ret = hipz_h_destroy_eq(shca->ipz_hca_handle, eq); - +@@ -169,7 +169,7 @@ int ehca_destroy_eq(struct ehca_shca *sh +unsigned long flags; +u64 h_ret; + +- ibmebus_free_irq(eq->ist, (void *)shca); ++ ibmebus_free_irq(NULL, eq->ist, (void *)shca); + +spin_lock_irqsave(&shca_list_lock, flags); +eq->is_initialized = 0; Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_main.c === --- ofa_kernel-1.5.orig/drivers/infiniband/hw/ehca/ehca_main.c 2009-09-25 12:42:48.0 +0200 diff --git a/kernel_patches/backport/2.6.16_sles10_sp3/ehca-030-ibmebus_loc_code.patch b/kernel_patches/backport/2.6.16_sles10_sp3/ehca-030-ibmebus_loc_code.patch index 9d6830d..cf9b5bb 100644 --- a/kernel_patches/backport/2.6.16_sles10_sp3/ehca-030-ibmebus_loc_code.patch +++ b/kernel_patches/backport/2.6.16_sles10_sp3/ehca-030-ibmebus_loc_code.patch @@ -39,15 +39,15 @@ Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_eq.c IRQF_DISABLED, "ehca_neq", (void *)shca); if (ret < 0) -@@ -170,7 +170,7 @@ - u64 h_ret; +@@ -169,7 +169,7 @@ +unsigned long flags; +u64 h_ret; - spin_lock_irqsave(&eq->spinlock, flags); -- ibmebus_free_irq(eq->ist, (void *)shca); -+ ibmebus_free_irq(NULL, eq->ist, (void *)shca); - - h_ret = hipz_h_destroy_eq(shca->ipz_hca_handle, eq); +- ibmebus_free_irq(eq->ist, (void *)shca); ++ ibmebus_free_irq(NULL, eq->ist, (void *)shca); +spin_lock_irqsave(&shca_list_lock, flags); +eq->is_initialized = 0; Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_main.c === --- ofa_kernel-1.5.orig/drivers/infiniband/hw/ehca/ehca_main.c 2009-09-25 12:42:48.0 +0200 diff --git a/kernel_patches/backport/2.6.18-EL5.3/ehca-030-ibmebus_loc_code.patch b/kernel_patches/backport/2.6.18-EL5.3/ehca-030-ibmebus_loc_code.patch index 9d6830d..cfc1595 100644 --- a/kernel_patches/backport/2.6.18-EL5.3/ehca-030-ibmebus_loc_code
[ewg] [PATCH OFED-151] ehca forward ports
Hi Vlad,
please apply for OFED-151.
Forward ports for ehca driver to enable compilation
on 2.6.32 and 2.6.31.
Signed-off-by: Alexander Schmidt
---
kernel_patches/backport/2.6.32/ehca-010-remove_driver_data.patch | 60
++
kernel_patches/backport/2.6.32/ehca-020-fix_buswalk.patch| 17 ++
2 files changed, 77 insertions(+)
--- /dev/null
+++
ofed_kernel-1.5/kernel_patches/backport/2.6.32/ehca-010-remove_driver_data.patch
@@ -0,0 +1,60 @@
+commit f899c2ddd45f2515deb446e2b143e4a686a49aee
+Author: Greg Kroah-Hartman
+Date: Mon May 4 12:40:54 2009 -0700
+
+infiniband: ehca: remove driver_data direct access of struct device
+
+In the near future, the driver core is going to not allow direct access
+to the driver_data pointer in struct device. Instead, the functions
+dev_get_drvdata() and dev_set_drvdata() should be used. These functions
+have been around since the beginning, so are backwards compatible with
+all older kernel versions.
+
+Cc: Sean Hefty
+Cc: Roland Dreier
+Cc: Hal Rosenstock
+Cc: gene...@lists.openfabrics.org
+Cc: Christoph Raisch
+Acked-by: Hoang-Nam Nguyen
+Signed-off-by: Greg Kroah-Hartman
+
+diff --git a/drivers/infiniband/hw/ehca/ehca_main.c
b/drivers/infiniband/hw/ehca/ehca_main.c
+index 85905ab..ce4e6ef 100644
+--- a/drivers/infiniband/hw/ehca/ehca_main.c
b/drivers/infiniband/hw/ehca/ehca_main.c
+@@ -636,7 +636,7 @@ static ssize_t ehca_show_##name(struct device *dev,
\
+ struct hipz_query_hca *rblock; \
+ int data; \
+ \
+- shca = dev->driver_data; \
++ shca = dev_get_drvdata(dev); \
+ \
+ rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL); \
+ if (!rblock) { \
+@@ -680,7 +680,7 @@ static ssize_t ehca_show_adapter_handle(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+ {
+- struct ehca_shca *shca = dev->driver_data;
++ struct ehca_shca *shca = dev_get_drvdata(dev);
+
+ return sprintf(buf, "%llx\n", shca->ipz_hca_handle.handle);
+
+@@ -749,7 +749,7 @@ static int __devinit ehca_probe(struct of_device *dev,
+
+ shca->ofdev = dev;
+ shca->ipz_hca_handle.handle = *handle;
+- dev->dev.driver_data = shca;
++ dev_set_drvdata(&dev->dev, shca);
+
+ ret = ehca_sense_attributes(shca);
+ if (ret < 0) {
+@@ -878,7 +878,7 @@ probe1:
+
+ static int __devexit ehca_remove(struct of_device *dev)
+ {
+- struct ehca_shca *shca = dev->dev.driver_data;
++ struct ehca_shca *shca = dev_get_drvdata(&dev->dev);
+ unsigned long flags;
+ int ret;
+
--- /dev/null
+++ ofed_kernel-1.5/kernel_patches/backport/2.6.32/ehca-020-fix_buswalk.patch
@@ -0,0 +1,17 @@
+---
+ drivers/infiniband/hw/ehca/ehca_mrmw.c |2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+Index: ofa_kernel-1.5.1/drivers/infiniband/hw/ehca/ehca_mrmw.c
+===
+--- ofa_kernel-1.5.1.orig/drivers/infiniband/hw/ehca/ehca_mrmw.c
ofa_kernel-1.5.1/drivers/infiniband/hw/ehca/ehca_mrmw.c
+@@ -2463,7 +2463,7 @@ int ehca_create_busmap(void)
+ int ret;
+
+ ehca_mr_len = 0;
+- ret = walk_memory_resource(0, 1ULL << MAX_PHYSMEM_BITS, NULL,
++ ret = walk_system_ram_range(0, 1ULL << MAX_PHYSMEM_BITS, NULL,
+ ehca_create_busmap_callback);
+ return ret;
+ }
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH OFED-151] ehca in install.pl
Hi Vlad,
another patch for OFED-1.5.1...
Signed-off-by: Alexander Schmidt
---
install.pl |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- OFED-1.5.1-20100217-0757.orig/install.pl
+++ OFED-1.5.1-20100217-0757/install.pl
@@ -1658,7 +1658,7 @@ sub set_availability
# Ehca
if ($arch =~ m/ppc64|powerpc/ and
-$kernel =~ m/2.6.1[6-9]|2.6.2[0-9]|2.6.30/) {
+$kernel =~ m/2.6.1[6-9]|2.6.2[0-9]|2.6.3[0-2]/) {
$kernel_modules_info{'ehca'}{'available'} = 1;
$packages_info{'libehca'}{'available'} = 1;
$packages_info{'libehca-devel-static'}{'available'} = 1;
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] [PATCH OFED-151] ehca forward ports
On Thu, 18 Feb 2010 19:27:14 +0200
Vladimir Sokolovsky wrote:
> Alexander Schmidt wrote:
> > Hi Vlad,
> >
> > please apply for OFED-151.
> >
> > Forward ports for ehca driver to enable compilation
> > on 2.6.32 and 2.6.31.
> >
> > Signed-off-by: Alexander Schmidt
> > ---
> > kernel_patches/backport/2.6.32/ehca-010-remove_driver_data.patch | 60
> > ++
> > kernel_patches/backport/2.6.32/ehca-020-fix_buswalk.patch| 17 ++
> > 2 files changed, 77 insertions(+)
> >
>
> Hi Alex,
> I don't see patches for 2.6.31. Should they be here?
Hi Vlad,
the comment was more intended for the install.pl change, sorry ;) For 2.6.31,
the code compiles without additional patches.
Regards,
Alex
>
> Regards,
> Vladimir
>
> > --- /dev/null
> > +++
> > ofed_kernel-1.5/kernel_patches/backport/2.6.32/ehca-010-remove_driver_data.patch
> > @@ -0,0 +1,60 @@
> > +commit f899c2ddd45f2515deb446e2b143e4a686a49aee
> > +Author: Greg Kroah-Hartman
> > +Date: Mon May 4 12:40:54 2009 -0700
> > +
> > +infiniband: ehca: remove driver_data direct access of struct device
> > +
> > +In the near future, the driver core is going to not allow direct access
> > +to the driver_data pointer in struct device. Instead, the functions
> > +dev_get_drvdata() and dev_set_drvdata() should be used. These
> > functions
> > +have been around since the beginning, so are backwards compatible with
> > +all older kernel versions.
> > +
> > +Cc: Sean Hefty
> > +Cc: Roland Dreier
> > +Cc: Hal Rosenstock
> > +Cc: gene...@lists.openfabrics.org
> > +Cc: Christoph Raisch
> > +Acked-by: Hoang-Nam Nguyen
> > +Signed-off-by: Greg Kroah-Hartman
> > +
> > +diff --git a/drivers/infiniband/hw/ehca/ehca_main.c
> > b/drivers/infiniband/hw/ehca/ehca_main.c
> > +index 85905ab..ce4e6ef 100644
> > +--- a/drivers/infiniband/hw/ehca/ehca_main.c
> > b/drivers/infiniband/hw/ehca/ehca_main.c
> > +@@ -636,7 +636,7 @@ static ssize_t ehca_show_##name(struct device *dev,
> > \
> > + struct hipz_query_hca *rblock; \
> > + int data; \
> > + \
> > +- shca = dev->driver_data; \
> > ++ shca = dev_get_drvdata(dev); \
> > + \
> > + rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL); \
> > + if (!rblock) { \
> > +@@ -680,7 +680,7 @@ static ssize_t ehca_show_adapter_handle(struct device
> > *dev,
> > + struct device_attribute *attr,
> > + char *buf)
> > + {
> > +- struct ehca_shca *shca = dev->driver_data;
> > ++ struct ehca_shca *shca = dev_get_drvdata(dev);
> > +
> > + return sprintf(buf, "%llx\n", shca->ipz_hca_handle.handle);
> > +
> > +@@ -749,7 +749,7 @@ static int __devinit ehca_probe(struct of_device *dev,
> > +
> > + shca->ofdev = dev;
> > + shca->ipz_hca_handle.handle = *handle;
> > +- dev->dev.driver_data = shca;
> > ++ dev_set_drvdata(&dev->dev, shca);
> > +
> > + ret = ehca_sense_attributes(shca);
> > + if (ret < 0) {
> > +@@ -878,7 +878,7 @@ probe1:
> > +
> > + static int __devexit ehca_remove(struct of_device *dev)
> > + {
> > +- struct ehca_shca *shca = dev->dev.driver_data;
> > ++ struct ehca_shca *shca = dev_get_drvdata(&dev->dev);
> > + unsigned long flags;
> > + int ret;
> > +
> > --- /dev/null
> > +++
> > ofed_kernel-1.5/kernel_patches/backport/2.6.32/ehca-020-fix_buswalk.patch
> > @@ -0,0 +1,17 @@
> > +---
> > + drivers/infiniband/hw/ehca/ehca_mrmw.c |2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +Index: ofa_kernel-1.5.1/drivers/infiniband/hw/ehca/ehca_mrmw.c
> > +===
> > +--- ofa_kernel-1.5.1.orig/drivers/infiniband/hw/ehca/ehca_mrmw.c
> > ofa_kernel-1.5.1/drivers/infiniband/hw/ehca/ehca_mrmw.c
> > +@@ -2463,7 +2463,7 @@ int ehca_create_busmap(void)
> > + int ret;
> > +
> > + ehca_mr_len = 0;
> > +- ret = walk_memory_resource(0, 1ULL << MAX_PHYSMEM_BITS, NULL,
> > ++ ret = walk_system_ram_range(0, 1ULL << MAX_PHYSMEM_BITS, NULL,
> > + ehca_create_busmap_callback);
> > + return ret;
> > + }
> >
>
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH OFED-151] ehca: bump version number
Hi Vlad,
please add this for OFED-1.5.1.
Signed-off-by: Alexander Schmidt
diff --git a/kernel_patches/fixes/ehca-0140-bump_version.patch
b/kernel_patches/fixes/ehca-0140-bump_version.patch
new file mode 100644
index 000..d28e073
--- /dev/null
+++ b/kernel_patches/fixes/ehca-0140-bump_version.patch
@@ -0,0 +1,13 @@
+Index: ofa_kernel-1.5.1/drivers/infiniband/hw/ehca/ehca_main.c
+===
+--- ofa_kernel-1.5.1.orig/drivers/infiniband/hw/ehca/ehca_main.c
ofa_kernel-1.5.1/drivers/infiniband/hw/ehca/ehca_main.c
+@@ -52,7 +52,7 @@
+ #include "ehca_tools.h"
+ #include "hcp_if.h"
+
+-#define HCAD_VERSION "0029"
++#define HCAD_VERSION "0030"
+
+ MODULE_LICENSE("Dual BSD/GPL");
+ MODULE_AUTHOR("Christoph Raisch ");
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] update release notes of all components for OFED 1.5.1
On Mon, 15 Mar 2010 17:31:57 -0700 Tziporet Koren wrote: > Hi All > > Since we wish to release OFED 1.5.1 very soon I wish to remind you to > update your component release notes if it have been changed from OFED 1.5 Hi Vlad, please apply these changes for the ehca release notes, thank you! diff --git a/ehca_release_notes.txt b/ehca_release_notes.txt index ad7f34a..3581f27 100644 --- a/ehca_release_notes.txt +++ b/ehca_release_notes.txt @@ -1,8 +1,8 @@ Open Fabrics Enterprise Distribution (OFED) -ehca in OFED 1.5 Release Notes +ehca in OFED 1.5.1 Release Notes - December 2009 + March 2010 Overview @@ -38,8 +38,13 @@ whereby is one of the following items: New Features -- DMEM toleration -- Port autodetection set to default +- None + +Fixed Bugs ofed-1.5.1 +- +- Fixed crash when reading sysfs performance counters +- Do not disable IRQs when processing EQs +- Allow query of max_dest_rd_atomic and max_qp_rd_atomic values Fixed Bugs ofed-1.5 - @@ -93,7 +98,7 @@ Available backports - SLES11: 2.6.27.19-5.1-smp - SLES10SP2: 2.6.16-60 - SLES10SP3: 2.6.16.60-0.54.5 -- kernel.org: 2.6.27-30 +- kernel.org: 2.6.29-32 Known Issues ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [RFC] libibverbs: ibv_fork_init() and libhugetlbfs
Hi all,
we are trying to make use of libhugetlbfs in an application that relies on
ibv_fork_init() to enable fork() support. The problem we are running into is
that calls to the madvise system call fail when registering a memory region
for memory that is provided by libhugetlbfs. We have written a preliminary
fix (see below) for this and are looking for comments / feedback to get an
acceptable solution.
When fork support is enabled in libibverbs, madvise() is called for every
memory page that is registered as a memory region. Memory ranges that
are passed to madvise() must be page aligned and the size must be a
multiple of the page size. libibverbs uses sysconf(_SC_PAGESIZE) to find
out the system page size and rounds all ranges passed to reg_mr() according
to this page size. When memory from libhugetlbfs is passed to reg_mr(), this
does not work as the page size for this memory range might be different
(e.g. 16Mb). So libibverbs would have to use the huge page size to
calculate a page aligned range for madvise.
As huge pages are provided to the application "under the hood" when
preloading libhugetlbfs, the application does not have any knowledge about
when it registers a huge page or a usual page.
The patch below demonstrates a possible solution for this. It parses the
/proc/PID/maps file when registering a memory region and decides if the
memory that is to be registered is part of a libhugetlbfs range or not. If so,
a page size of 16Mb is used to align the memory range passed to madvise().
We see two problems with this: it is not a very elegant solution to parse the
procfs file and the 16Mb are hardcoded currently. The latter point could be
solved by calling gethugepagesize() from libhugetlbfs, which would add a new
dependency to libibverbs.
We are highly interested in reviews, comments, suggestions to get this solved
soon. Thanks!
Signed-off-by: Alexander Schmidt
---
src/memory.c | 50 +++---
1 file changed, 47 insertions(+), 3 deletions(-)
--- libibverbs-1.1.2.orig/src/memory.c
+++ libibverbs-1.1.2/src/memory.c
@@ -40,6 +40,8 @@
#include
#include
#include
+#include
+#include
#include "ibverbs.h"
@@ -54,6 +56,8 @@
#define MADV_DOFORK11
#endif
+#define HUGE_PAGE_SIZE (16 * 1024 * 1024)
+
struct ibv_mem_node {
enum {
IBV_RED,
@@ -446,6 +450,48 @@ static struct ibv_mem_node *__mm_find_st
return node;
}
+static void get_range_address(uintptr_t *start, uintptr_t *end, void *base,
size_t size)
+{
+ pid_t pid;
+ FILE *file;
+ char buf[1024], lib[128];
+ int range_page_size = page_size;
+
+ pid = getpid();
+ snprintf(buf, sizeof(buf), "/proc/%d/maps", pid);
+
+ file = fopen(buf, "r");
+ if (!file)
+ goto out;
+
+ while (fgets(buf, sizeof(buf), file) != NULL) {
+ int n;
+ char *substr;
+ uintptr_t range_start, range_end;
+
+ n = sscanf(buf, "%lx-%lx %*s %*x %*s %*u %127s",
+ &range_start, &range_end, &lib);
+
+ if (n < 3)
+ continue;
+
+ substr = strstr(lib, "libhugetlbfs");
+ if (substr) {
+ if ((uintptr_t) base >= range_start &&
+ (uintptr_t) base < range_end) {
+ range_page_size = HUGE_PAGE_SIZE;
+ break;
+ }
+ }
+ }
+ fclose(file);
+
+out:
+ *start = (uintptr_t) base & ~(range_page_size - 1);
+ *end = ((uintptr_t) (base + size + range_page_size - 1) &
+~(range_page_size - 1)) - 1;
+}
+
static int ibv_madvise_range(void *base, size_t size, int advice)
{
uintptr_t start, end;
@@ -458,9 +504,7 @@ static int ibv_madvise_range(void *base,
inc = advice == MADV_DONTFORK ? 1 : -1;
- start = (uintptr_t) base & ~(page_size - 1);
- end = ((uintptr_t) (base + size + page_size - 1) &
-~(page_size - 1)) - 1;
+ get_range_address(&start, &end, base, size);
pthread_mutex_lock(&mm_mutex);
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] [RFC] libibverbs: ibv_fork_init() and libhugetlbfs
On Thu, 06 May 2010 13:55:31 -0700
Roland Dreier wrote:
> I think that we cannot assume huge pages only come from libhugetlbfs --
> we should support an application directly enabling huge pages (possibly
> via another library too, so we can't assume that an application knows
> the page size for a memory range it is about to register).
>
> And also the 16 MB page size constant is of course not feasible -- with
> all due respect, the x86 page size of 2 MB is much more likely in
> practice :) (Although perhaps the much slower PowerPC TLB refill makes
> users more likely to try and use hugetlb pages ;)
>
> Alex suggested parsing files in the same way as libhugetlbfs does to get
> the page size, and that seems to be the best solution, since I don't
> think the libhugetlbfs license is compatible with the BSD license for
> libibverbs.
>
> But your trick of using /proc/*/maps looks nice. Does that only work
> for libhugetlbfs or can we recognize direct mmap of hugetlb pages?
Hi Roland, thanks for your comments!
I've reworked my patch:
* added get_huge_page_size() to read the huge page size from
/proc/meminfo. This is done at ibv_fork_init() time.
* I noticed that some applications like ibv_rc_pingpong already
get memory from libhugetlbfs when running ibv_fork_init(). So
I changed the code for testing madvise() to allocate a huge page
if the huge page size is set in the system.
I have not tested this code with different libraries providing huge
pages / mmaped pages yet, but I hope this can be added later on when
we have agreed on an approach to handle huge pages.
Signed-off-by: Alexander Schmidt
---
src/memory.c | 103 ++-
1 file changed, 95 insertions(+), 8 deletions(-)
--- libibverbs-1.1.2.orig/src/memory.c
+++ libibverbs-1.1.2/src/memory.c
@@ -40,6 +40,8 @@
#include
#include
#include
+#include
+#include
#include "ibverbs.h"
@@ -68,12 +70,45 @@ struct ibv_mem_node {
static struct ibv_mem_node *mm_root;
static pthread_mutex_t mm_mutex = PTHREAD_MUTEX_INITIALIZER;
static int page_size;
+static int huge_page_size;
static int too_late;
+static int get_huge_page_size(void)
+{
+ int ret = -1;
+ FILE *file;
+ char *path = "/proc/meminfo";
+ char buf[1024], type[128];
+
+ file = fopen(path, "r");
+ if (!file)
+ goto out;
+
+ while (fgets(buf, sizeof(buf), file) != NULL) {
+ int n;
+ unsigned long size;
+
+ n = sscanf(buf, "%127s %lu %*s", &type, &size);
+
+ if (n < 2)
+ continue;
+
+ if (!strcmp(type, "Hugepagesize:")) {
+ /* huge page size is printed in Kb */
+ ret = size * 1024;
+ break;
+ }
+ }
+ fclose(file);
+
+out:
+ return ret;
+}
+
int ibv_fork_init(void)
{
void *tmp;
- int ret;
+ int ret, size;
if (mm_root)
return 0;
@@ -85,11 +120,18 @@ int ibv_fork_init(void)
if (page_size < 0)
return errno;
- if (posix_memalign(&tmp, page_size, page_size))
+ huge_page_size = get_huge_page_size();
+
+ if (huge_page_size > page_size)
+ size = huge_page_size;
+ else
+ size = page_size;
+
+ if (posix_memalign(&tmp, size, size))
return ENOMEM;
- ret = madvise(tmp, page_size, MADV_DONTFORK) ||
- madvise(tmp, page_size, MADV_DOFORK);
+ ret = madvise(tmp, size, MADV_DONTFORK) ||
+ madvise(tmp, size, MADV_DOFORK);
free(tmp);
@@ -446,11 +488,51 @@ static struct ibv_mem_node *__mm_find_st
return node;
}
+static int is_huge_page(void *base)
+{
+ int ret = 0;
+ pid_t pid;
+ FILE *file;
+ char buf[1024], lib[128];
+
+ pid = getpid();
+ snprintf(buf, sizeof(buf), "/proc/%d/maps", pid);
+
+ file = fopen(buf, "r");
+ if (!file)
+ goto out;
+
+ while (fgets(buf, sizeof(buf), file) != NULL) {
+ int n;
+ char *substr;
+ uintptr_t range_start, range_end;
+
+ n = sscanf(buf, "%lx-%lx %*s %*x %*s %*u %127s",
+ &range_start, &range_end, &lib);
+
+ if (n < 3)
+ continue;
+
+ substr = strstr(lib, "libhugetlbfs");
+ if (substr) {
+ if ((uintptr_t) base >= range_start &&
+ (uintptr_t) base < range_end) {
+ ret = 1;
+ break;
+ }
+
[ewg] New libehca tarball
Hi Vlad, please include the new libehca tarball for OFED-1.5.2: http://www.openfabrics.org/downloads/libehca/libehca-1.2.1-0.2.g68c8294.tar.gz It contains the following changes: Stefan Roscher (1): libehca: Reject UD send work requests if they exceed the MTU size. Kind regards, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH v2] libibverbs: ibv_fork_init() and libhugetlbfs
Hi Roland,
we have been working on adressing your review comments and are looking for
feedback regarding v2 now.
Problem description:
When fork support is enabled in libibverbs, madvise() is called for every
memory page that is registered as a memory region. Memory ranges that
are passed to madvise() must be page aligned and the size must be a
multiple of the page size. libibverbs uses sysconf(_SC_PAGESIZE) to find
out the system page size and rounds all ranges passed to reg_mr() according
to this page size. When memory from libhugetlbfs is passed to reg_mr(), this
does not work as the page size for this memory range might be different
(e.g. 16Mb). So libibverbs would have to use the huge page size to
calculate a page aligned range for madvise.
As huge pages are provided to the application "under the hood" when
preloading libhugetlbfs, the application does not have any knowledge about
when it registers a huge page or a usual page.
To work around this issue, detect the use of huge pages in libibverbs and
align memory ranges passed to madvise according to the huge page size.
Changes since v1:
- detect use of huge pages at ibv_fork_init() time by walking through
/sys/kernel/mm/hugepages/
- read huge page size from /proc/pid/smaps, which contains the page
size of the mapping (thereby enabling support for mutliple huge
page sizes)
- code is independent of libhugetlbfs now, so huge pages can be provided
to the application by any library
Performance:
PPC64 system with eHCA
without patch:
1M memory region120usec
16M memory region 1970usec
with patch v2:
1M memory region172usec
16M memory region 2030usec
with patch and 16M huge pages:
1M memory region110usec
16M memory region 193usec
Signed-off-by: Alexander Schmidt
---
src/memory.c | 137 ---
1 file changed, 131 insertions(+), 6 deletions(-)
--- libibverbs-1.1.2.orig/src/memory.c
+++ libibverbs-1.1.2/src/memory.c
@@ -40,6 +40,10 @@
#include
#include
#include
+#include
+#include
+#include
+#include
#include "ibverbs.h"
@@ -68,12 +72,117 @@ struct ibv_mem_node {
static struct ibv_mem_node *mm_root;
static pthread_mutex_t mm_mutex = PTHREAD_MUTEX_INITIALIZER;
static int page_size;
+static int huge_page_enabled;
static int too_late;
+static int is_huge_page_enabled(void)
+{
+ int n, ret = 0;
+ char *bufp;
+ DIR *dir;
+ struct dirent *entry;
+ FILE *file;
+ unsigned long nr_hugepages;
+ char buf[1024];
+
+ dir = opendir("/sys/kernel/mm/hugepages/");
+ if (!dir)
+ return 0;
+
+ while ((entry = readdir(dir))) {
+ if (strncmp(entry->d_name, "hugepages-", 10))
+ continue;
+
+ snprintf(buf, sizeof(buf),
"/sys/kernel/mm/hugepages/%s/nr_hugepages",
+ entry->d_name);
+
+ file = fopen(buf, "r");
+ if (!file)
+ continue;
+
+ bufp = fgets(buf, sizeof(buf), file);
+ fclose(file);
+ if (!bufp)
+ continue;
+
+ n = sscanf(buf, "%lu", &nr_hugepages);
+ if (n < 1)
+ continue;
+
+ if (nr_hugepages) {
+ ret = 1;
+ goto out;
+ }
+ }
+
+out:
+ closedir(dir);
+
+ return ret;
+}
+
+static unsigned long smaps_page_size(FILE *file)
+{
+ int n;
+ unsigned long size = page_size;
+ char buf[1024];
+
+ while (fgets(buf, sizeof(buf), file) != NULL) {
+ if (!strstr(buf, "KernelPageSize:"))
+ continue;
+
+ n = sscanf(buf, "%*s %lu", &size);
+ if (n < 1)
+ continue;
+
+ /* page size is printed in Kb */
+ size = size * 1024;
+
+ break;
+ }
+
+ return size;
+}
+
+static unsigned long get_page_size(void *base)
+{
+ unsigned long ret = page_size;
+ pid_t pid;
+ FILE *file;
+ char buf[1024];
+
+ pid = getpid();
+ snprintf(buf, sizeof(buf), "/proc/%d/smaps", pid);
+
+ file = fopen(buf, "r");
+ if (!file)
+ goto out;
+
+ while (fgets(buf, sizeof(buf), file) != NULL) {
+ int n;
+ uintptr_t range_start, range_end;
+
+ n = sscanf(buf, "%lx-%lx", &range_start, &range_end);
+
+ if (n < 2)
+ continue;
+
+ if ((uintptr_t) base >= range_start && (uintptr_t) base <
range_end) {
+ ret = smaps_page_size(file);
+ break;
+ }
+ }
+ fclose(file);
+
+ou
Re: [ewg] [PATCH v2] libibverbs: ibv_fork_init() and libhugetlbfs
On Wed, 02 Jun 2010 14:49:37 -0700
Roland Dreier wrote:
> So if I read this correctly this patch introduces almost a 50% overhead
> in the 1M case... and probably much worse (as a fraction) in say the 64K
> or 4K case. I wonder if that's acceptable?
We don't think this is acceptable, so we like the third approach you suggested
very much. I've written the code and attached it below. This third version
does not introduce additional overhead when not using huge pages (verified
with 4k, 64k, 1m and 16m memory regions).
Problem description:
When fork support is enabled in libibverbs, madvise() is called for every
memory page that is registered as a memory region. Memory ranges that
are passed to madvise() must be page aligned and the size must be a
multiple of the page size. libibverbs uses sysconf(_SC_PAGESIZE) to find
out the system page size and rounds all ranges passed to reg_mr() according
to this page size. When memory from libhugetlbfs is passed to reg_mr(), this
does not work as the page size for this memory range might be different
(e.g. 16Mb). So libibverbs would have to use the huge page size to
calculate a page aligned range for madvise.
As huge pages are provided to the application "under the hood" when
preloading libhugetlbfs, the application does not have any knowledge about
when it registers a huge page or a usual page.
To work around this issue, detect the use of huge pages in libibverbs and
align memory ranges passed to madvise according to the huge page size.
Changes since v1:
- detect use of huge pages at ibv_fork_init() time by walking through
/sys/kernel/mm/hugepages/
- read huge page size from /proc/pid/smaps, which contains the page
size of the mapping (thereby enabling support for mutliple huge
page sizes)
- code is independent of libhugetlbfs now, so huge pages can be provided
to the application by any library
Changes since v2:
- reading from /proc/ to determine the huge page size is now only done
when a call to madvise() using the system page size fails. So there
is no overhead introduced when registering non-huge-page memory.
Signed-off-by: Alexander Schmidt
---
src/memory.c | 96 +++
1 file changed, 90 insertions(+), 6 deletions(-)
--- libibverbs.git.orig/src/memory.c
+++ libibverbs.git/src/memory.c
@@ -40,6 +40,8 @@
#include
#include
#include
+#include
+#include
#include "ibverbs.h"
@@ -70,10 +72,64 @@ static pthread_mutex_t mm_mutex = PTHREA
static int page_size;
static int too_late;
+static unsigned long smaps_page_size(FILE *file)
+{
+ int n;
+ unsigned long size = 0;
+ char buf[1024];
+
+ while (fgets(buf, sizeof(buf), file) != NULL) {
+ if (!strstr(buf, "KernelPageSize:"))
+ continue;
+
+ n = sscanf(buf, "%*s %lu", &size);
+ if (n < 1)
+ continue;
+
+ /* page size is printed in Kb */
+ size = size * 1024;
+
+ break;
+ }
+
+ return size;
+}
+
+static unsigned long get_page_size(void *base)
+{
+ unsigned long ret = 0;
+ FILE *file;
+ char buf[1024];
+
+ file = fopen("/proc/self/smaps", "r");
+ if (!file)
+ goto out;
+
+ while (fgets(buf, sizeof(buf), file) != NULL) {
+ int n;
+ uintptr_t range_start, range_end;
+
+ n = sscanf(buf, "%lx-%lx", &range_start, &range_end);
+
+ if (n < 2)
+ continue;
+
+ if ((uintptr_t) base >= range_start && (uintptr_t) base <
range_end) {
+ ret = smaps_page_size(file);
+ break;
+ }
+ }
+ fclose(file);
+
+out:
+ return ret;
+}
+
int ibv_fork_init(void)
{
- void *tmp;
+ void *tmp, *tmp_aligned;
int ret;
+ unsigned long size;
if (mm_root)
return 0;
@@ -88,8 +144,17 @@ int ibv_fork_init(void)
if (posix_memalign(&tmp, page_size, page_size))
return ENOMEM;
- ret = madvise(tmp, page_size, MADV_DONTFORK) ||
- madvise(tmp, page_size, MADV_DOFORK);
+ size = get_page_size(tmp);
+
+ if (size)
+ tmp_aligned = (void *)((uintptr_t)tmp & ~(size - 1));
+ else {
+ size = page_size;
+ tmp_aligned = tmp;
+ }
+
+ ret = madvise(tmp_aligned, size, MADV_DONTFORK) ||
+ madvise(tmp_aligned, size, MADV_DOFORK);
free(tmp);
@@ -522,7 +587,8 @@ static struct ibv_mem_node *undo_node(st
return node;
}
-static int ibv_madvise_range(void *base, size_t size, int advice)
+static int ibv_madvise_range(void *base, size_t size, int advice,
+unsign
[ewg] [PATCH OFED-152] ib/ehca: bitmaks handling for lock_hcalls
Hi Vlad, please add this fix for OFED-1.5.2, thank you! Signed-off-by: ALexander Schmidt Index: ofa_kernel-1.5.2/kernel_patches/fixes/ehca-0150-lock_hcalls_bitmask.patch === --- /dev/null 1970-01-01 00:00:00.0 + +++ ofa_kernel-1.5.2/kernel_patches/fixes/ehca-0150-lock_hcalls_bitmask.patch 2010-06-11 04:50:46.0 -0400 @@ -0,0 +1,22 @@ +Fix reading capability bit from device capabilities. + +Signed-off-by: Alexander Schmidt +--- + + drivers/infiniband/hw/ehca/ehca_main.c |3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +Index: ofa_kernel-1.5.2/drivers/infiniband/hw/ehca/ehca_main.c +=== +--- ofa_kernel-1.5.2.orig/drivers/infiniband/hw/ehca/ehca_main.c 2010-06-11 04:03:56.0 -0400 ofa_kernel-1.5.2/drivers/infiniband/hw/ehca/ehca_main.c2010-06-11 04:47:47.0 -0400 +@@ -359,7 +359,8 @@ +* a firmware property, so it's valid across all adapters +*/ + if (ehca_lock_hcalls == -1) +- ehca_lock_hcalls = !(shca->hca_cap & HCA_CAP_H_ALLOC_RES_SYNC); ++ ehca_lock_hcalls = !EHCA_BMASK_GET(HCA_CAP_H_ALLOC_RES_SYNC, ++ shca->hca_cap); + + /* translate supported MR page sizes; always support 4K */ + shca->hca_cap_mr_pgsize = EHCA_PAGESIZE; ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] OFED-1.5.2 32bit packaging
Hi Vlad, I'm running into an issue with 32bit packages on SLES11SP1 (ppc64, daily build OFED-1.5.2-20100607-0636). During the build, 32bit and 64bit libibverbs packages are generated which are stored as RPMS/sles-release-11.1-1.152/ppc64/libibverbs-1.1.3-0.8.g4d733f4.ppc64.rpm RPMS/sles-release-11.1-1.152/ppc64/libibverbs-1.1.3-0.8.g4d733f4.ppc.rpm Then the installer tries to install both, which works fine for the 64bit package but installing the 32bit version fails with Preparing packages for installation... package libibverbs-1.1.3-0.8.g4d733f4.ppc is already installed I've double checked that there aren't any old packages installed, seems like the 64bit and 32bit version are colliding. Do you have an idea what could be the reason for this? If not I'll try to investigate this. Thanks! ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED-1.5.2 32bit packaging
On Tue, 15 Jun 2010 17:09:24 +0300
Vladimir Sokolovsky wrote:
> Hi Alexander,
> Please try OFED installation after applying the following patch to the
> install.pl script:
>
>
> diff --git a/install.pl b/install.pl
> index 89ca93c..54f3b0f 100755
> --- a/install.pl
> +++ b/install.pl
> @@ -3481,7 +3481,7 @@ sub install_rpm_32
> }
>
> $cmd = "rpm -iv $rpminstall_flags";
> -if ($distro eq "SuSE" and $dist_rpm_rel gt 15.2) {
> +if ($distro eq "SuSE") {
> $cmd .= " --force";
> }
> $cmd .= " $package";
Hi Vlad,
this worked fine on SLES11SP1, will this be the final fix or could this break
other (older) SuSE releases?
Anyway, thanks for looking into this!
Regards,
Alex
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] [PATCH v2] libibverbs: ibv_fork_init() and libhugetlbfs
On Thu, 10 Jun 2010 17:59:28 +0300 Alex Vainman wrote: > Wrote Roland Dreier: > > Thanks, nice work. I like this approach. Alex (Vainman) any comments > > on this? > > > > - R. > > The solution looks great. Hi all, in our further testing, we noticed that there is a substantial problem with the current solution. Depending on the order of memory registrations, we might end up with a corrupted node tree which blocks regions from being registered. When registering two memory regions A and B from within the same huge page, we will end up with one node in the tree which covers the whole huge page after registering A. When the second MR is registered, a node is created with the MR size rounded to the system page size (as there is no need to call madvise(), it is not noticed that MR B is part of a huge page). Now if MR A is deregistered before MR B, I see that the tree containing mem_nodes is empty afterwards, which causes problems for the deregistration of MR B, leaving the tree in a corrupted state with negative refcounts. This also breaks later registrations of other memory regions within this huge page. At the moment I do not see an obvious solution for this, but it's clear that an overhaul of this code is needed. I'm writing this to make sure that there won't be a release of libibverbs containing this incomplete code, but also to ask for comments from other people who might have an idea on how to fix this. Thanks for any comments! Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] [PATCH v2] libibverbs: ibv_fork_init() and libhugetlbfs
On Sat, 03 Jul 2010 13:19:07 -0700 Roland Dreier wrote: > > When registering two memory regions A and B from within > > the same huge page, we will end up with one node in the tree which covers > the > > whole huge page after registering A. When the second MR is registered, a > node > > is created with the MR size rounded to the system page size (as there is no > > need to call madvise(), it is not noticed that MR B is part of a huge > page). > > > > Now if MR A is deregistered before MR B, I see that the tree containing > > mem_nodes is empty afterwards, which causes problems for the > deregistration of > > MR B, leaving the tree in a corrupted state with negative refcounts. This > also > > breaks later registrations of other memory regions within this huge page. > > Good thing I didn't get around to applying the patch yet ;) > > I haven't thought this through fully, but it seems that maybe we could > extend the madvise tracking tree to keep track of the page size used for > each node in the tree. Then for the registration of MR B above, we > would find the node for MR A covered MR B and we should be able to get > the ref counting right. We thought about this too, but in some special cases, we do not know the correct page size of a memory range. For example when getting a 16M chunk from a 16M huge page region which is also aligned to 16M, the first madvise() will work fine and the code will assume that the page size is 64K. If trying to register a 16M - 64K + 1 byte region, the first madvise() also works fine. Now if a second memory region which resides in the last 64K is registered, we end up with the same situation as above. As this issue was not present in version 2 of the code, but there we had a big performance penalty, I suggest the following: we could go back to version 2 and introduce a new RDMAV_HUGEPAGE_SAFE env variable to let the user decide between huge page support and better performance (the same approach we use for the COW protection itself). Would this be okay or do you see a problem with this? Regards, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH v4] libibverbs: ibv_fork_init() and huge pages
Hi Roland,
this is the fourth version of the huge pages support, please see below for
change log + description, we appreciate your comments.
Problem description:
When fork support is enabled in libibverbs, madvise() is called for every
memory page that is registered as a memory region. Memory ranges that
are passed to madvise() must be page aligned and the size must be a
multiple of the page size. libibverbs uses sysconf(_SC_PAGESIZE) to find
out the system page size and rounds all ranges passed to reg_mr() according
to this page size. When memory from libhugetlbfs is passed to reg_mr(), this
does not work as the page size for this memory range might be different
(e.g. 16Mb). So libibverbs would have to use the huge page size to
calculate a page aligned range for madvise.
As huge pages are provided to the application "under the hood" when
preloading libhugetlbfs, the application does not have any knowledge about
when it registers a huge page or a usual page.
To work around this issue, detect the use of huge pages in libibverbs and
align memory ranges passed to madvise according to the huge page size.
Changes since v1:
- detect use of huge pages at ibv_fork_init() time by walking through
/sys/kernel/mm/hugepages/
- read huge page size from /proc/pid/smaps, which contains the page
size of the mapping (thereby enabling support for mutliple huge
page sizes)
- code is independent of libhugetlbfs now, so huge pages can be provided
to the application by any library
Changes since v2:
- reading from /proc/ to determine the huge page size is now only done
when a call to madvise() using the system page size fails. So there
is no overhead introduced when registering non-huge-page memory.
Changes since v3:
- determining the page size of a given memory range by watching madvise()
fail has proven to be unreliable. So we introduce the RDMAV_HUGEPAGES_SAFE
environment variable to let the user decide if the page size should be
checked on every reg_mr() call or not. This requires the user to be aware
if huge pages are used by the running application or not.
I did not add an aditional API call to enable this, as applications can use
setenv() + ibv_fork_init() to enable checking for huge pages in the code.
Signed-off-by: Alexander Schmidt
---
src/memory.c | 94 +++
1 file changed, 88 insertions(+), 6 deletions(-)
--- libibverbs.git.orig/src/memory.c
+++ libibverbs.git/src/memory.c
@@ -40,6 +40,10 @@
#include
#include
#include
+#include
+#include
+#include
+#include
#include "ibverbs.h"
@@ -68,12 +72,71 @@ struct ibv_mem_node {
static struct ibv_mem_node *mm_root;
static pthread_mutex_t mm_mutex = PTHREAD_MUTEX_INITIALIZER;
static int page_size;
+static int huge_page_enabled;
static int too_late;
+static unsigned long smaps_page_size(FILE *file)
+{
+ int n;
+ unsigned long size = page_size;
+ char buf[1024];
+
+ while (fgets(buf, sizeof(buf), file) != NULL) {
+ if (!strstr(buf, "KernelPageSize:"))
+ continue;
+
+ n = sscanf(buf, "%*s %lu", &size);
+ if (n < 1)
+ continue;
+
+ /* page size is printed in Kb */
+ size = size * 1024;
+
+ break;
+ }
+
+ return size;
+}
+
+static unsigned long get_page_size(void *base)
+{
+ unsigned long ret = page_size;
+ pid_t pid;
+ FILE *file;
+ char buf[1024];
+
+ pid = getpid();
+ snprintf(buf, sizeof(buf), "/proc/%d/smaps", pid);
+
+ file = fopen(buf, "r");
+ if (!file)
+ goto out;
+
+ while (fgets(buf, sizeof(buf), file) != NULL) {
+ int n;
+ uintptr_t range_start, range_end;
+
+ n = sscanf(buf, "%lx-%lx", &range_start, &range_end);
+
+ if (n < 2)
+ continue;
+
+ if ((uintptr_t) base >= range_start && (uintptr_t) base <
range_end) {
+ ret = smaps_page_size(file);
+ break;
+ }
+ }
+ fclose(file);
+
+out:
+ return ret;
+}
+
int ibv_fork_init(void)
{
- void *tmp;
+ void *tmp, *tmp_aligned;
int ret;
+ unsigned long size;
if (mm_root)
return 0;
@@ -88,8 +151,21 @@ int ibv_fork_init(void)
if (posix_memalign(&tmp, page_size, page_size))
return ENOMEM;
- ret = madvise(tmp, page_size, MADV_DONTFORK) ||
- madvise(tmp, page_size, MADV_DOFORK);
+ if (getenv("RDMAV_HUGEPAGES_SAFE"))
+ huge_page_enabled = 1;
+ else
+ huge_page_enabled = 0;
+
+ if (huge_page_enabled) {
+ size = get_page_size(tmp);
+ tmp_aligned = (v
Re: [ewg] [PATCH v4] libibverbs: ibv_fork_init() and huge pages
On Mon, 19 Jul 2010 13:34:29 +0200
Alexander Schmidt wrote:
> Hi Roland,
>
> this is the fourth version of the huge pages support, please see below for
> change log + description, we appreciate your comments.
Hi Roland,
did you have a chance to take a look at the code? We would like to get this
into upstream libibverbs, so any comments are appreciated.
Regards,
Alex
>
> Problem description:
>
> When fork support is enabled in libibverbs, madvise() is called for every
> memory page that is registered as a memory region. Memory ranges that
> are passed to madvise() must be page aligned and the size must be a
> multiple of the page size. libibverbs uses sysconf(_SC_PAGESIZE) to find
> out the system page size and rounds all ranges passed to reg_mr() according
> to this page size. When memory from libhugetlbfs is passed to reg_mr(), this
> does not work as the page size for this memory range might be different
> (e.g. 16Mb). So libibverbs would have to use the huge page size to
> calculate a page aligned range for madvise.
>
> As huge pages are provided to the application "under the hood" when
> preloading libhugetlbfs, the application does not have any knowledge about
> when it registers a huge page or a usual page.
>
> To work around this issue, detect the use of huge pages in libibverbs and
> align memory ranges passed to madvise according to the huge page size.
>
> Changes since v1:
>
> - detect use of huge pages at ibv_fork_init() time by walking through
> /sys/kernel/mm/hugepages/
> - read huge page size from /proc/pid/smaps, which contains the page
> size of the mapping (thereby enabling support for mutliple huge
> page sizes)
> - code is independent of libhugetlbfs now, so huge pages can be provided
> to the application by any library
>
> Changes since v2:
>
> - reading from /proc/ to determine the huge page size is now only done
> when a call to madvise() using the system page size fails. So there
> is no overhead introduced when registering non-huge-page memory.
>
> Changes since v3:
>
> - determining the page size of a given memory range by watching madvise()
> fail has proven to be unreliable. So we introduce the RDMAV_HUGEPAGES_SAFE
> environment variable to let the user decide if the page size should be
> checked on every reg_mr() call or not. This requires the user to be aware
> if huge pages are used by the running application or not.
>
> I did not add an aditional API call to enable this, as applications can use
> setenv() + ibv_fork_init() to enable checking for huge pages in the code.
>
> Signed-off-by: Alexander Schmidt
> ---
> src/memory.c | 94
> +++
> 1 file changed, 88 insertions(+), 6 deletions(-)
>
> --- libibverbs.git.orig/src/memory.c
> +++ libibverbs.git/src/memory.c
> @@ -40,6 +40,10 @@
> #include
> #include
> #include
> +#include
> +#include
> +#include
> +#include
>
> #include "ibverbs.h"
>
> @@ -68,12 +72,71 @@ struct ibv_mem_node {
> static struct ibv_mem_node *mm_root;
> static pthread_mutex_t mm_mutex = PTHREAD_MUTEX_INITIALIZER;
> static int page_size;
> +static int huge_page_enabled;
> static int too_late;
>
> +static unsigned long smaps_page_size(FILE *file)
> +{
> + int n;
> + unsigned long size = page_size;
> + char buf[1024];
> +
> + while (fgets(buf, sizeof(buf), file) != NULL) {
> + if (!strstr(buf, "KernelPageSize:"))
> + continue;
> +
> + n = sscanf(buf, "%*s %lu", &size);
> + if (n < 1)
> + continue;
> +
> + /* page size is printed in Kb */
> + size = size * 1024;
> +
> + break;
> + }
> +
> + return size;
> +}
> +
> +static unsigned long get_page_size(void *base)
> +{
> + unsigned long ret = page_size;
> + pid_t pid;
> + FILE *file;
> + char buf[1024];
> +
> + pid = getpid();
> + snprintf(buf, sizeof(buf), "/proc/%d/smaps", pid);
> +
> + file = fopen(buf, "r");
> + if (!file)
> + goto out;
> +
> + while (fgets(buf, sizeof(buf), file) != NULL) {
> + int n;
> + uintptr_t range_start, range_end;
> +
> + n = sscanf(buf, "%lx-%lx", &range_start, &range_end);
> +
> + if (n < 2)
> + continue;
> +
> + if ((uintptr_t) base >= range_start && (uintptr_t) base <
> range_end) {
> + ret = smaps_page_size(
[ewg] [PATCH OFED-153] ehca: fixes
Hi Vlad,
please apply the following fixes + backport adjustments for OFED-1.5.3.
The changes are tested with ./ofed_scripts/ofed_makedist.sh
Signed-off-by: Alexander Schmidt
diff --git
a/kernel_patches/backport/2.6.16_sles10_sp2/ehca-030-ibmebus_loc_code.patch
b/kernel_patches/backport/2.6.16_sles10_sp2/ehca-030-ibmebus_loc_code.patch
index 23a7f1e..2fea541 100644
--- a/kernel_patches/backport/2.6.16_sles10_sp2/ehca-030-ibmebus_loc_code.patch
+++ b/kernel_patches/backport/2.6.16_sles10_sp2/ehca-030-ibmebus_loc_code.patch
@@ -4,10 +4,10 @@
drivers/infiniband/hw/ehca/ehca_main.c| 49
++
3 files changed, 27 insertions(+), 30 deletions(-)
-Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_classes.h
+Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/hw/ehca/ehca_classes.h
===
ofa_kernel-1.5.orig/drivers/infiniband/hw/ehca/ehca_classes.h
2009-09-23 15:08:25.0 +0200
-+++ ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_classes.h 2009-09-25
12:43:39.0 +0200
+---
ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/hw/ehca/ehca_classes.h
2011-01-03 05:15:26.0 -0500
ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/hw/ehca/ehca_classes.h
2011-01-03 05:20:26.0 -0500
@@ -112,7 +112,7 @@
struct ehca_shca {
@@ -17,41 +17,41 @@ Index:
ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_classes.h
u8 num_ports;
int hw_level;
struct list_head shca_list;
-Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_eq.c
+Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/hw/ehca/ehca_eq.c
===
ofa_kernel-1.5.orig/drivers/infiniband/hw/ehca/ehca_eq.c 2009-09-23
15:08:25.0 +0200
-+++ ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_eq.c2009-09-25
12:43:39.0 +0200
-@@ -122,7 +122,7 @@
-
- /* register interrupt handlers and initialize work queues */
+--- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/hw/ehca/ehca_eq.c
2011-01-03 05:15:27.0 -0500
ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/hw/ehca/ehca_eq.c
2011-01-03 05:21:46.0 -0500
+@@ -124,7 +124,7 @@
if (type == EHCA_EQ) {
+ tasklet_init(&eq->interrupt_task, ehca_tasklet_eq, (long)shca);
+
- ret = ibmebus_request_irq(eq->ist, ehca_interrupt_eq,
+ ret = ibmebus_request_irq(NULL, eq->ist, ehca_interrupt_eq,
IRQF_DISABLED, "ehca_eq",
(void *)shca);
if (ret < 0)
-@@ -130,7 +130,7 @@
-
- tasklet_init(&eq->interrupt_task, ehca_tasklet_eq, (long)shca);
+@@ -132,7 +132,7 @@
} else if (type == EHCA_NEQ) {
+ tasklet_init(&eq->interrupt_task, ehca_tasklet_neq, (long)shca);
+
- ret = ibmebus_request_irq(eq->ist, ehca_interrupt_neq,
+ ret = ibmebus_request_irq(NULL, eq->ist, ehca_interrupt_neq,
IRQF_DISABLED, "ehca_neq",
(void *)shca);
if (ret < 0)
-@@ -169,7 +169,7 @@ int ehca_destroy_eq(struct ehca_shca *sh
-unsigned long flags;
-u64 h_ret;
-
-- ibmebus_free_irq(eq->ist, (void *)shca);
-+ ibmebus_free_irq(NULL, eq->ist, (void *)shca);
-
-spin_lock_irqsave(&shca_list_lock, flags);
-eq->is_initialized = 0;
-Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_main.c
+@@ -169,7 +169,7 @@
+ unsigned long flags;
+ u64 h_ret;
+
+- ibmebus_free_irq(eq->ist, (void *)shca);
++ ibmebus_free_irq(NULL, eq->ist, (void *)shca);
+
+ spin_lock_irqsave(&shca_list_lock, flags);
+ eq->is_initialized = 0;
+Index: ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/hw/ehca/ehca_main.c
===
ofa_kernel-1.5.orig/drivers/infiniband/hw/ehca/ehca_main.c 2009-09-25
12:42:48.0 +0200
-+++ ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_main.c 2009-09-25
12:45:40.0 +0200
+--- ofed_kernel-2.6.16_sles10_sp2.orig/drivers/infiniband/hw/ehca/ehca_main.c
2011-01-03 05:16:09.0 -0500
ofed_kernel-2.6.16_sles10_sp2/drivers/infiniband/hw/ehca/ehca_main.c
2011-01-03 05:20:26.0 -0500
@@ -289,8 +289,8 @@
};
@@ -63,7 +63,7 @@ Index: ofa_kernel-1.5/drivers/infiniband/hw/ehca/ehca_main.c
if (loc_code)
ehca_gen_dbg(" ... location lode=%s", loc_code);
-@@ -459,7 +459,7 @@
+@@ -460,7 +460,7 @@
shca->ib_device.node_type = RDMA_NODE_IB_CA;
shca->ib_device.phys_port_cnt = shca->num_ports;
shca->ib_dev
[ewg] OFED-1.5.3-rc1 compile problem in mlx4
Hi Tziporet, on a PowerPC system with SLES11 SP1 I'm facing the following compile problem with OFED-1.5.3-rc1, can you or someone from your team have a look at this? We appreciate your help. Regards, Alex -I/usr/src/linux-2.6.32.12-0.7/arch//include \ -Iinclude \ \ -I/usr/src/linux-2.6.32.12-0.7/arch/powerpc/include \ -D__KERNEL__ -Iarch/powerpc -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O2 -msoft-float -pipe -Iarch/powerpc -mminimal-toc -mtraceback=none -mcall-aixdesc -mcpu=power4 -mtune=cell -mno-altivec -mno-spe -mspe=no -funit-at-a-time -mno-string -Wa,-maltivec -fno-stack-protector -fomit-frame-pointer -g -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(main)" -D"KBUILD_MODNAME=KBUILD_STR(mlx4_core)" -D"DEBUG_HASH=38" -D"DEBUG_HASH2=51" -c -o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.3/drivers/net/mlx4/.tmp_main.o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.3/drivers/net/mlx4/main.c In file included from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.3/drivers/net/mlx4/main.c:41: include/linux/io-mapping.h:24:23: error: asm/iomap.h: No such file or directory make[3]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.3/drivers/net/mlx4/main.o] Error 1 make[2]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.3/drivers/net/mlx4] Error 2 make[1]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.3] Error 2 make[1]: Leaving directory `/usr/src/linux-2.6.32.12-0.7' make: *** [kernel] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.52194 (%build) ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] docs: update ehca release notes for OFED-1.5.3
Hi Vlad, please apply the following changes to the docs git. Regards, Alex Signed-off-by: Alexander Schmidt diff --git a/release_notes/ehca_release_notes.txt b/release_notes/ehca_release_notes.txt index e1ca30b..02dc246 100644 --- a/release_notes/ehca_release_notes.txt +++ b/release_notes/ehca_release_notes.txt @@ -1,8 +1,8 @@ Open Fabrics Enterprise Distribution (OFED) -ehca in OFED 1.5.2 Release Notes +ehca in OFED 1.5.3 Release Notes -September 2010 +February 2011 Overview @@ -40,6 +40,11 @@ New Features - None +Fixed Bugs ofed-1.5.3 +- +- Fixed initialization of IRQ tasklet +- Catch fails in ioremap() + Fixed Bugs ofed-1.5.2 - - Fixed automatic detection if hcall locks should be enabled or not @@ -97,12 +102,11 @@ Fixed Bugs ofed-1.3 Available backports --- -- RedHat EL5 up4: 2.6.18-164.ELsmp - RedHat EL5 up5: 2.6.18-194.ELsmp -- SLES11: 2.6.27.19-5.1-smp +- RedHat EL6: 2.6.32-71.el6 - SLES11SP1: 2.6.32.12-0.7-default - SLES10SP3: 2.6.16.60-0.54.5 -- kernel.org: 2.6.29-32 +- kernel.org: 2.6.30-36 Known Issues ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] EWG/OFED bi-weekly engineering teleconf - change in time for tomorrow
On Sun, 27 Mar 2011 09:53:31 + Tziporet Koren wrote: > Topic: OFED bi-weekly engineering teleconf > Date: Every 2 weeks on Monday > Time: 12pm, EST; 9am PST; 7pm Israel > > Toll Free US dial in number is 1-888-875-9370 > Israel local dial in number is +972-8-612-5663 > > Woodi – can you please send the bridge info > Is there any dial in information available for the call today? thx, Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH OFED-154] ehca: backports for RHEL-6.1
Hi Vlad,
please apply the following backport for OFED-1.5.4.
Signed-off-by: Alexander Schmidt
Index:
ofa_kernel-1.5.4/kernel_patches/backport/2.6.32-EL6.1/ehca-010-remove_driver_data.patch
===
--- /dev/null 1970-01-01 00:00:00.0 +
+++
ofa_kernel-1.5.4/kernel_patches/backport/2.6.32-EL6.1/ehca-010-remove_driver_data.patch
2011-11-21 18:09:25.0 +0100
@@ -0,0 +1,60 @@
+commit f899c2ddd45f2515deb446e2b143e4a686a49aee
+Author: Greg Kroah-Hartman
+Date: Mon May 4 12:40:54 2009 -0700
+
+infiniband: ehca: remove driver_data direct access of struct device
+
+In the near future, the driver core is going to not allow direct access
+to the driver_data pointer in struct device. Instead, the functions
+dev_get_drvdata() and dev_set_drvdata() should be used. These functions
+have been around since the beginning, so are backwards compatible with
+all older kernel versions.
+
+Cc: Sean Hefty
+Cc: Roland Dreier
+Cc: Hal Rosenstock
+Cc: gene...@lists.openfabrics.org
+Cc: Christoph Raisch
+Acked-by: Hoang-Nam Nguyen
+Signed-off-by: Greg Kroah-Hartman
+
+diff --git a/drivers/infiniband/hw/ehca/ehca_main.c
b/drivers/infiniband/hw/ehca/ehca_main.c
+index 85905ab..ce4e6ef 100644
+--- a/drivers/infiniband/hw/ehca/ehca_main.c
b/drivers/infiniband/hw/ehca/ehca_main.c
+@@ -636,7 +636,7 @@ static ssize_t ehca_show_##name(struct device *dev,
\
+ struct hipz_query_hca *rblock; \
+ int data; \
+ \
+- shca = dev->driver_data; \
++ shca = dev_get_drvdata(dev); \
+ \
+ rblock = ehca_alloc_fw_ctrlblock(GFP_KERNEL); \
+ if (!rblock) { \
+@@ -680,7 +680,7 @@ static ssize_t ehca_show_adapter_handle(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+ {
+- struct ehca_shca *shca = dev->driver_data;
++ struct ehca_shca *shca = dev_get_drvdata(dev);
+
+ return sprintf(buf, "%llx\n", shca->ipz_hca_handle.handle);
+
+@@ -749,7 +749,7 @@ static int __devinit ehca_probe(struct of_device *dev,
+
+ shca->ofdev = dev;
+ shca->ipz_hca_handle.handle = *handle;
+- dev->dev.driver_data = shca;
++ dev_set_drvdata(&dev->dev, shca);
+
+ ret = ehca_sense_attributes(shca);
+ if (ret < 0) {
+@@ -878,7 +878,7 @@ probe1:
+
+ static int __devexit ehca_remove(struct of_device *dev)
+ {
+- struct ehca_shca *shca = dev->dev.driver_data;
++ struct ehca_shca *shca = dev_get_drvdata(&dev->dev);
+ unsigned long flags;
+ int ret;
+
Index:
ofa_kernel-1.5.4/kernel_patches/backport/2.6.32-EL6.1/ehca-020-fix_buswalk.patch
===
--- /dev/null 1970-01-01 00:00:00.0 +
+++
ofa_kernel-1.5.4/kernel_patches/backport/2.6.32-EL6.1/ehca-020-fix_buswalk.patch
2011-11-21 18:10:02.0 +0100
@@ -0,0 +1,17 @@
+---
+ drivers/infiniband/hw/ehca/ehca_mrmw.c |2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+Index: ofa_kernel-1.5.1/drivers/infiniband/hw/ehca/ehca_mrmw.c
+===
+--- ofa_kernel-1.5.1.orig/drivers/infiniband/hw/ehca/ehca_mrmw.c
ofa_kernel-1.5.1/drivers/infiniband/hw/ehca/ehca_mrmw.c
+@@ -2463,7 +2463,7 @@ int ehca_create_busmap(void)
+ int ret;
+
+ ehca_mr_len = 0;
+- ret = walk_memory_resource(0, 1ULL << MAX_PHYSMEM_BITS, NULL,
++ ret = walk_system_ram_range(0, 1ULL << MAX_PHYSMEM_BITS, NULL,
+ ehca_create_busmap_callback);
+ return ret;
+ }
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 2/5] ib/ehca: rename goto label
Rename the "poll_cq_one_read_cqe" goto label to what it actually does,
"repoll".
Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]>
---
drivers/infiniband/hw/ehca/ehca_reqs.c |6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_reqs.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_reqs.c
@@ -589,7 +589,7 @@ static inline int ehca_poll_cq_one(struc
struct ehca_qp *my_qp;
int cqe_count = 0, is_error;
-poll_cq_one_read_cqe:
+repoll:
cqe = (struct ehca_cqe *)
ipz_qeit_get_inc_valid(&my_cq->ipz_queue);
if (!cqe) {
@@ -617,7 +617,7 @@ poll_cq_one_read_cqe:
ehca_dmp(cqe, 64, "cq_num=%x qp_num=%x",
my_cq->cq_number,
cqe->local_qp_number); /* ignore this purged cqe */
- goto poll_cq_one_read_cqe;
+ goto repoll;
}
spin_lock_irqsave(&qp->spinlock_s, flags);
purgeflag = qp->sqerr_purgeflag;
@@ -636,7 +636,7 @@ poll_cq_one_read_cqe:
* that caused sqe and turn off purge flag
*/
qp->sqerr_purgeflag = 0;
- goto poll_cq_one_read_cqe;
+ goto repoll;
}
}
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 3/5] ib/ehca: repoll on invalid opcode
When the ehca driver detects an invalid opcode in a CQE, it currently passes the CQE to the application and returns with success. This patch changes the CQE handling to discard CQEs with invalid opcodes and to continue reading the next CQE from the CQ. Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]> --- drivers/infiniband/hw/ehca/ehca_reqs.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_reqs.c +++ infiniband.git/drivers/infiniband/hw/ehca/ehca_reqs.c @@ -667,7 +667,7 @@ repoll: ehca_dmp(cqe, 64, "ehca_cq=%p cq_num=%x", my_cq, my_cq->cq_number); /* update also queue adder to throw away this entry!!! */ - goto poll_cq_one_exit0; + goto repoll; } /* eval ib_wc_status */ ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 1/5] ib/ehca: update qp_state on cached modify_qp()
Since the introduction of the port auto-detect mode for ehca, calls to
modify_qp() may be cached in the device driver when the ports are not
activated yet. When a modify_qp() call is cached, the qp state remains
untouched until the port is activated, which will leave the qp in the
reset state. In the reset state, however, it is not allowed to post SQ
WQEs, which confuses applications like ib_mad.
The solution for this problem is to immediately set the qp state as
requested by modify_qp(), even when the call is cached.
Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]>
---
drivers/infiniband/hw/ehca/ehca_qp.c | 14 ++
1 file changed, 10 insertions(+), 4 deletions(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_qp.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_qp.c
@@ -1534,8 +1534,6 @@ static int internal_modify_qp(struct ib_
if (attr_mask & IB_QP_QKEY)
my_qp->qkey = attr->qkey;
- my_qp->state = qp_new_state;
-
modify_qp_exit2:
if (squeue_locked) { /* this means: sqe -> rts */
spin_unlock_irqrestore(&my_qp->spinlock_s, flags);
@@ -1551,6 +1549,8 @@ modify_qp_exit1:
int ehca_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int
attr_mask, struct ib_udata *udata)
{
+ int ret = 0;
+
struct ehca_shca *shca = container_of(ibqp->device, struct
ehca_shca, ib_device);
struct ehca_qp *my_qp = container_of(ibqp, struct ehca_qp,
ib_qp); @@ -1597,12 +1597,18 @@ int ehca_modify_qp(struct ib_qp *ibqp, s
attr->qp_state,
my_qp->init_attr.port_num, ibqp->qp_type);
spin_unlock_irqrestore(&sport->mod_sqp_lock,
flags);
- return 0;
+ goto out;
}
spin_unlock_irqrestore(&sport->mod_sqp_lock, flags);
}
- return internal_modify_qp(ibqp, attr, attr_mask, 0);
+ ret = internal_modify_qp(ibqp, attr, attr_mask, 0);
+
+out:
+ if ((ret == 0) && (attr_mask & IB_QP_STATE))
+ my_qp->state = attr->qp_state;
+
+ return ret;
}
void ehca_recover_sqp(struct ib_qp *sqp)
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 4/5] ib/ehca: check idr_find() return value
The idr_find() function may fail when trying to get the QP that is associated with a CQE, e.g. when a QP has been destroyed between the generation of a CQE and the poll request for it. In consequence, the return value of idr_find() must be checked and the CQE must be discarded when the QP cannot be found. Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]> --- drivers/infiniband/hw/ehca/ehca_reqs.c |4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) --- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_reqs.c +++ infiniband.git/drivers/infiniband/hw/ehca/ehca_reqs.c @@ -680,8 +680,10 @@ repoll: read_lock(&ehca_qp_idr_lock); my_qp = idr_find(&ehca_qp_idr, cqe->qp_token); - wc->qp = &my_qp->ib_qp; read_unlock(&ehca_qp_idr_lock); + if (!my_qp) + goto repoll; + wc->qp = &my_qp->ib_qp; wc->byte_len = cqe->nr_bytes_transferred; wc->pkey_index = cqe->pkey_index; ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 5/5] ib/ehca: discard double CQE for one WR
Under rare circumstances, the ehca hardware might erroneously generate
two CQEs for the same WQE, which is not compliant to the IB spec and
will cause unpredictable errors like memory being freed twice. To avoid
this problem, the driver needs to detect the second CQE and discard it.
For this purpose, introduce an array holding as many elements as the SQ
of the QP, called sq_map. Each sq_map entry stores a "reported" flag
for one WQE in the SQ. When a work request is posted to the SQ, the
respective "reported" flag is set to zero. After the arrival of a CQE,
the flag is set to 1, which allows to detect the occurence of a second
CQE.
The mapping between WQE / CQE and the corresponding sq_map element is
implemented by replacing the lowest 16 Bits of the wr_id with the index
in the queue map. The original 16 Bits are stored in the sq_map entry
and are restored when the CQE is passed to the application.
Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]>
---
drivers/infiniband/hw/ehca/ehca_classes.h |9 +
drivers/infiniband/hw/ehca/ehca_qes.h |1
drivers/infiniband/hw/ehca/ehca_qp.c | 34 +-
drivers/infiniband/hw/ehca/ehca_reqs.c| 54
+++--- 4 files changed, 78 insertions(+), 20
deletions(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_reqs.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_reqs.c
@@ -139,6 +139,7 @@ static void trace_send_wr_ud(const struc
static inline int ehca_write_swqe(struct ehca_qp *qp,
struct ehca_wqe *wqe_p,
const struct ib_send_wr *send_wr,
+ u32 sq_map_idx,
int hidden)
{
u32 idx;
@@ -157,7 +158,11 @@ static inline int ehca_write_swqe(struct
/* clear wqe header until sglist */
memset(wqe_p, 0, offsetof(struct ehca_wqe, u.ud_av.sg_list));
- wqe_p->work_request_id = send_wr->wr_id;
+ wqe_p->work_request_id = send_wr->wr_id & ~QMAP_IDX_MASK;
+ wqe_p->work_request_id |= sq_map_idx & QMAP_IDX_MASK;
+
+ qp->sq_map[sq_map_idx].app_wr_id = send_wr->wr_id &
QMAP_IDX_MASK;
+ qp->sq_map[sq_map_idx].reported = 0;
switch (send_wr->opcode) {
case IB_WR_SEND:
@@ -381,6 +386,7 @@ static inline int post_one_send(struct e
{
struct ehca_wqe *wqe_p;
int ret;
+ u32 sq_map_idx;
u64 start_offset = my_qp->ipz_squeue.current_q_offset;
/* get pointer next to free WQE */
@@ -393,8 +399,15 @@ static inline int post_one_send(struct e
"qp_num=%x", my_qp->ib_qp.qp_num);
return -ENOMEM;
}
+
+ /*
+* Get the index of the WQE in the send queue. The same index
is used
+* for writing into the sq_map.
+*/
+ sq_map_idx = start_offset / my_qp->ipz_squeue.qe_size;
+
/* write a SEND WQE into the QUEUE */
- ret = ehca_write_swqe(my_qp, wqe_p, cur_send_wr, hidden);
+ ret = ehca_write_swqe(my_qp, wqe_p, cur_send_wr, sq_map_idx,
hidden); /*
* if something failed,
* reset the free entry pointer to the start value
@@ -654,8 +667,34 @@ repoll:
my_cq, my_cq->cq_number);
}
- /* we got a completion! */
- wc->wr_id = cqe->work_request_id;
+ read_lock(&ehca_qp_idr_lock);
+ my_qp = idr_find(&ehca_qp_idr, cqe->qp_token);
+ read_unlock(&ehca_qp_idr_lock);
+ if (!my_qp)
+ goto repoll;
+ wc->qp = &my_qp->ib_qp;
+
+ if (!(cqe->w_completion_flags & WC_SEND_RECEIVE_BIT)) {
+ struct ehca_qmap_entry *qmap_entry;
+ /*
+* We got a send completion and need to restore the
original
+* wr_id.
+*/
+ qmap_entry = &my_qp->sq_map[cqe->work_request_id &
+ QMAP_IDX_MASK];
+
+ if (qmap_entry->reported) {
+ ehca_warn(cq->device, "Double cqe on
qp_num=%#x",
+ my_qp->real_qp_num);
+ /* found a double cqe, discard it and read
next one */
+ goto repoll;
+ }
+ wc->wr_id = cqe->work_request_id & ~QMAP_IDX_MASK;
+ wc->wr_id |= qmap_entry->app_wr_id;
+ qmap_entry->reported = 1;
+ } else
+ /* We got a receive completion. */
+ wc->wr_id = cqe->work_request_id;
/* eval ib_wc_opcode */
wc->opcode = ib_wc_opcode[cqe->optype]-1;
@@ -678,13 +717,6 @@ repoll:
} else
wc->status = IB_WC_SUCCESS;
- read_lock(&ehca_qp_idr_lock);
- my_qp = idr_find(&
[ewg] [PATCH 0/5] ib/ehca: Fix stability issues
Hi Roland, the following patchset contains four small fixes and one bigger patch (5/5) for addressing some ehca issues we found during cluster test. [1/5] update qp_state on cached modify_qp() [2/5] rename goto label in ehca_poll_cq_one() [3/5] repoll on invalid opcode instead of returning success [4/5] check idr_find() return value [5/5] discard double CQE for one WR They all apply on top of 2.6.27-rc1. If possible, we would like to get them into 2.6.27. Regards, Alexander Schmidt ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 1/5 try2] ib/ehca: update qp_state on cached modify_qp()
Since the introduction of the port auto-detect mode for ehca, calls to
modify_qp() may be cached in the device driver when the ports are not
activated yet. When a modify_qp() call is cached, the qp state remains
untouched until the port is activated, which will leave the qp in the reset
state. In the reset state, however, it is not allowed to post SQ WQEs,
which confuses applications like ib_mad.
The solution for this problem is to immediately set the qp state as requested
by modify_qp(), even when the call is cached.
Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]>
---
drivers/infiniband/hw/ehca/ehca_qp.c | 14 ++
1 file changed, 10 insertions(+), 4 deletions(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_qp.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_qp.c
@@ -1534,8 +1534,6 @@ static int internal_modify_qp(struct ib_
if (attr_mask & IB_QP_QKEY)
my_qp->qkey = attr->qkey;
- my_qp->state = qp_new_state;
-
modify_qp_exit2:
if (squeue_locked) { /* this means: sqe -> rts */
spin_unlock_irqrestore(&my_qp->spinlock_s, flags);
@@ -1551,6 +1549,8 @@ modify_qp_exit1:
int ehca_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask,
struct ib_udata *udata)
{
+ int ret = 0;
+
struct ehca_shca *shca = container_of(ibqp->device, struct ehca_shca,
ib_device);
struct ehca_qp *my_qp = container_of(ibqp, struct ehca_qp, ib_qp);
@@ -1597,12 +1597,18 @@ int ehca_modify_qp(struct ib_qp *ibqp, s
attr->qp_state, my_qp->init_attr.port_num,
ibqp->qp_type);
spin_unlock_irqrestore(&sport->mod_sqp_lock, flags);
- return 0;
+ goto out;
}
spin_unlock_irqrestore(&sport->mod_sqp_lock, flags);
}
- return internal_modify_qp(ibqp, attr, attr_mask, 0);
+ ret = internal_modify_qp(ibqp, attr, attr_mask, 0);
+
+out:
+ if ((ret == 0) && (attr_mask & IB_QP_STATE))
+ my_qp->state = attr->qp_state;
+
+ return ret;
}
void ehca_recover_sqp(struct ib_qp *sqp)
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 3/5 try2] ib/ehca: repoll on invalid opcode
When the ehca driver detects an invalid opcode in a CQE, it currently passes the CQE to the application and returns with success. This patch changes the CQE handling to discard CQEs with invalid opcodes and to continue reading the next CQE from the CQ. Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]> --- drivers/infiniband/hw/ehca/ehca_reqs.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_reqs.c +++ infiniband.git/drivers/infiniband/hw/ehca/ehca_reqs.c @@ -667,7 +667,7 @@ repoll: ehca_dmp(cqe, 64, "ehca_cq=%p cq_num=%x", my_cq, my_cq->cq_number); /* update also queue adder to throw away this entry!!! */ - goto poll_cq_one_exit0; + goto repoll; } /* eval ib_wc_status */ ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 2/5 try2] ib/ehca: rename goto label
Rename the "poll_cq_one_read_cqe" goto label to what it actually does,
"repoll".
Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]>
---
drivers/infiniband/hw/ehca/ehca_reqs.c |6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_reqs.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_reqs.c
@@ -589,7 +589,7 @@ static inline int ehca_poll_cq_one(struc
struct ehca_qp *my_qp;
int cqe_count = 0, is_error;
-poll_cq_one_read_cqe:
+repoll:
cqe = (struct ehca_cqe *)
ipz_qeit_get_inc_valid(&my_cq->ipz_queue);
if (!cqe) {
@@ -617,7 +617,7 @@ poll_cq_one_read_cqe:
ehca_dmp(cqe, 64, "cq_num=%x qp_num=%x",
my_cq->cq_number, cqe->local_qp_number);
/* ignore this purged cqe */
- goto poll_cq_one_read_cqe;
+ goto repoll;
}
spin_lock_irqsave(&qp->spinlock_s, flags);
purgeflag = qp->sqerr_purgeflag;
@@ -636,7 +636,7 @@ poll_cq_one_read_cqe:
* that caused sqe and turn off purge flag
*/
qp->sqerr_purgeflag = 0;
- goto poll_cq_one_read_cqe;
+ goto repoll;
}
}
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 5/5 try2] ib/ehca: discard double CQE for one WR
Under rare circumstances, the ehca hardware might erroneously generate
two CQEs for the same WQE, which is not compliant to the IB spec and will
cause unpredictable errors like memory being freed twice. To avoid this
problem, the driver needs to detect the second CQE and discard it.
For this purpose, introduce an array holding as many elements as the SQ of
the QP, called sq_map. Each sq_map entry stores a "reported" flag for one
WQE in the SQ. When a work request is posted to the SQ, the respective
"reported" flag is set to zero. After the arrival of a CQE, the flag is set
to 1, which allows to detect the occurence of a second CQE.
The mapping between WQE / CQE and the corresponding sq_map element is
implemented by replacing the lowest 16 Bits of the wr_id with the index in
the queue map. The original 16 Bits are stored in the sq_map entry and are
restored when the CQE is passed to the application.
Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]>
---
drivers/infiniband/hw/ehca/ehca_classes.h |9 +
drivers/infiniband/hw/ehca/ehca_qes.h |1
drivers/infiniband/hw/ehca/ehca_qp.c | 34 +-
drivers/infiniband/hw/ehca/ehca_reqs.c| 54 +++---
4 files changed, 78 insertions(+), 20 deletions(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_reqs.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_reqs.c
@@ -139,6 +139,7 @@ static void trace_send_wr_ud(const struc
static inline int ehca_write_swqe(struct ehca_qp *qp,
struct ehca_wqe *wqe_p,
const struct ib_send_wr *send_wr,
+ u32 sq_map_idx,
int hidden)
{
u32 idx;
@@ -157,7 +158,11 @@ static inline int ehca_write_swqe(struct
/* clear wqe header until sglist */
memset(wqe_p, 0, offsetof(struct ehca_wqe, u.ud_av.sg_list));
- wqe_p->work_request_id = send_wr->wr_id;
+ wqe_p->work_request_id = send_wr->wr_id & ~QMAP_IDX_MASK;
+ wqe_p->work_request_id |= sq_map_idx & QMAP_IDX_MASK;
+
+ qp->sq_map[sq_map_idx].app_wr_id = send_wr->wr_id & QMAP_IDX_MASK;
+ qp->sq_map[sq_map_idx].reported = 0;
switch (send_wr->opcode) {
case IB_WR_SEND:
@@ -381,6 +386,7 @@ static inline int post_one_send(struct e
{
struct ehca_wqe *wqe_p;
int ret;
+ u32 sq_map_idx;
u64 start_offset = my_qp->ipz_squeue.current_q_offset;
/* get pointer next to free WQE */
@@ -393,8 +399,15 @@ static inline int post_one_send(struct e
"qp_num=%x", my_qp->ib_qp.qp_num);
return -ENOMEM;
}
+
+ /*
+* Get the index of the WQE in the send queue. The same index is used
+* for writing into the sq_map.
+*/
+ sq_map_idx = start_offset / my_qp->ipz_squeue.qe_size;
+
/* write a SEND WQE into the QUEUE */
- ret = ehca_write_swqe(my_qp, wqe_p, cur_send_wr, hidden);
+ ret = ehca_write_swqe(my_qp, wqe_p, cur_send_wr, sq_map_idx, hidden);
/*
* if something failed,
* reset the free entry pointer to the start value
@@ -654,8 +667,34 @@ repoll:
my_cq, my_cq->cq_number);
}
- /* we got a completion! */
- wc->wr_id = cqe->work_request_id;
+ read_lock(&ehca_qp_idr_lock);
+ my_qp = idr_find(&ehca_qp_idr, cqe->qp_token);
+ read_unlock(&ehca_qp_idr_lock);
+ if (!my_qp)
+ goto repoll;
+ wc->qp = &my_qp->ib_qp;
+
+ if (!(cqe->w_completion_flags & WC_SEND_RECEIVE_BIT)) {
+ struct ehca_qmap_entry *qmap_entry;
+ /*
+* We got a send completion and need to restore the original
+* wr_id.
+*/
+ qmap_entry = &my_qp->sq_map[cqe->work_request_id &
+ QMAP_IDX_MASK];
+
+ if (qmap_entry->reported) {
+ ehca_warn(cq->device, "Double cqe on qp_num=%#x",
+ my_qp->real_qp_num);
+ /* found a double cqe, discard it and read next one */
+ goto repoll;
+ }
+ wc->wr_id = cqe->work_request_id & ~QMAP_IDX_MASK;
+ wc->wr_id |= qmap_entry->app_wr_id;
+ qmap_entry->reported = 1;
+ } else
+ /* We got a receive completion. */
+ wc->wr_id = cqe->work_request_id;
/* eval ib_wc_opcode */
wc->opcode = ib_wc_opcode[cqe->optype]-1;
@@ -678,13 +717,6 @@ repoll:
} else
wc->status = IB_WC_SUCCESS;
- read_lock(&ehca_qp_idr_lock);
-
[ewg] [PATCH 4/5 try2] ib/ehca: check idr_find() return value
The idr_find() function may fail when trying to get the QP that is associated with a CQE, e.g. when a QP has been destroyed between the generation of a CQE and the poll request for it. In consequence, the return value of idr_find() must be checked and the CQE must be discarded when the QP cannot be found. Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]> --- drivers/infiniband/hw/ehca/ehca_reqs.c |4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) --- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_reqs.c +++ infiniband.git/drivers/infiniband/hw/ehca/ehca_reqs.c @@ -680,8 +680,10 @@ repoll: read_lock(&ehca_qp_idr_lock); my_qp = idr_find(&ehca_qp_idr, cqe->qp_token); - wc->qp = &my_qp->ib_qp; read_unlock(&ehca_qp_idr_lock); + if (!my_qp) + goto repoll; + wc->qp = &my_qp->ib_qp; wc->byte_len = cqe->nr_bytes_transferred; wc->pkey_index = cqe->pkey_index; ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH 0/5 try2] ib/ehca: Fix stability issues
Hi Roland, Sorry, the first set was mangled because of a broken mailer, so here it is again, double checked... the following patchset contains four small fixes and one bigger patch (5/5) for addressing some ehca issues we found during cluster test. [1/5] update qp_state on cached modify_qp() [2/5] rename goto label in ehca_poll_cq_one() [3/5] repoll on invalid opcode instead of returning success [4/5] check idr_find() return value [5/5] discard double CQE for one WR They all apply on top of 2.6.27-rc1. If possible, we would like to get them into 2.6.27. Regards, Alexander Schmidt ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [PATCH] ib/ehca: add flush CQE generation
When a QP goes into error state, it is required that flush CQEs are
delivered to the application for any outstanding work requests. eHCA does not
do this in hardware, so this patch adds software flush CQE generation to the
ehca driver.
Whenever a QP gets into error state, it is added to the QP error list of its
respective CQ. If the error QP list of a CQ is not empty, poll_cq()
generates flush CQEs before polling the actual CQ.
Signed-off-by: Alexander Schmidt <[EMAIL PROTECTED]>
---
Applies on top of 2.6.27-rc3, please consider this for 2.6.28.
drivers/infiniband/hw/ehca/ehca_classes.h | 14 +
drivers/infiniband/hw/ehca/ehca_cq.c |3
drivers/infiniband/hw/ehca/ehca_iverbs.h |2
drivers/infiniband/hw/ehca/ehca_qp.c | 225 --
drivers/infiniband/hw/ehca/ehca_reqs.c| 211
5 files changed, 412 insertions(+), 43 deletions(-)
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_classes.h
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_classes.h
@@ -164,6 +164,13 @@ struct ehca_qmap_entry {
u16 reported;
};
+struct ehca_queue_map {
+ struct ehca_qmap_entry *map;
+ unsigned int entries;
+ unsigned int tail;
+ unsigned int left_to_poll;
+};
+
struct ehca_qp {
union {
struct ib_qp ib_qp;
@@ -173,8 +180,9 @@ struct ehca_qp {
enum ehca_ext_qp_type ext_type;
enum ib_qp_state state;
struct ipz_queue ipz_squeue;
- struct ehca_qmap_entry *sq_map;
+ struct ehca_queue_map sq_map;
struct ipz_queue ipz_rqueue;
+ struct ehca_queue_map rq_map;
struct h_galpas galpas;
u32 qkey;
u32 real_qp_num;
@@ -204,6 +212,8 @@ struct ehca_qp {
atomic_t nr_events; /* events seen */
wait_queue_head_t wait_completion;
int mig_armed;
+ struct list_head sq_err_node;
+ struct list_head rq_err_node;
};
#define IS_SRQ(qp) (qp->ext_type == EQPT_SRQ)
@@ -233,6 +243,8 @@ struct ehca_cq {
/* mmap counter for resources mapped into user space */
u32 mm_count_queue;
u32 mm_count_galpa;
+ struct list_head sqp_err_list;
+ struct list_head rqp_err_list;
};
enum ehca_mr_flag {
--- infiniband.git.orig/drivers/infiniband/hw/ehca/ehca_reqs.c
+++ infiniband.git/drivers/infiniband/hw/ehca/ehca_reqs.c
@@ -53,9 +53,25 @@
/* in RC traffic, insert an empty RDMA READ every this many packets */
#define ACK_CIRC_THRESHOLD 200
+static u64 replace_wr_id(u64 wr_id, u16 idx)
+{
+ u64 ret;
+
+ ret = wr_id & ~QMAP_IDX_MASK;
+ ret |= idx & QMAP_IDX_MASK;
+
+ return ret;
+}
+
+static u16 get_app_wr_id(u64 wr_id)
+{
+ return wr_id & QMAP_IDX_MASK;
+}
+
static inline int ehca_write_rwqe(struct ipz_queue *ipz_rqueue,
struct ehca_wqe *wqe_p,
- struct ib_recv_wr *recv_wr)
+ struct ib_recv_wr *recv_wr,
+ u32 rq_map_idx)
{
u8 cnt_ds;
if (unlikely((recv_wr->num_sge < 0) ||
@@ -69,7 +85,7 @@ static inline int ehca_write_rwqe(struct
/* clear wqe header until sglist */
memset(wqe_p, 0, offsetof(struct ehca_wqe, u.ud_av.sg_list));
- wqe_p->work_request_id = recv_wr->wr_id;
+ wqe_p->work_request_id = replace_wr_id(recv_wr->wr_id, rq_map_idx);
wqe_p->nr_of_data_seg = recv_wr->num_sge;
for (cnt_ds = 0; cnt_ds < recv_wr->num_sge; cnt_ds++) {
@@ -146,6 +162,7 @@ static inline int ehca_write_swqe(struct
u64 dma_length;
struct ehca_av *my_av;
u32 remote_qkey = send_wr->wr.ud.remote_qkey;
+ struct ehca_qmap_entry *qmap_entry = &qp->sq_map.map[sq_map_idx];
if (unlikely((send_wr->num_sge < 0) ||
(send_wr->num_sge > qp->ipz_squeue.act_nr_of_sg))) {
@@ -158,11 +175,10 @@ static inline int ehca_write_swqe(struct
/* clear wqe header until sglist */
memset(wqe_p, 0, offsetof(struct ehca_wqe, u.ud_av.sg_list));
- wqe_p->work_request_id = send_wr->wr_id & ~QMAP_IDX_MASK;
- wqe_p->work_request_id |= sq_map_idx & QMAP_IDX_MASK;
+ wqe_p->work_request_id = replace_wr_id(send_wr->wr_id, sq_map_idx);
- qp->sq_map[sq_map_idx].app_wr_id = send_wr->wr_id & QMAP_IDX_MASK;
- qp->sq_map[sq_map_idx].reported = 0;
+ qmap_entry->app_wr_id = get_app_wr_id(send_wr->wr_id);
+ qmap_entry->reported = 0;
switch (send_wr->opcode) {
case IB_WR_SEND:
@@ -496,7 +512,9 @@ static int internal_post_recv(struct ehc
struct ehca_wqe *wqe_p;
int wqe_cnt = 0;
int ret = 0;
+ u32 rq_map_idx;
unsigned long flags;
+ struct ehca_qmap_entry *qmap_entry;
if (unlikely(!HAS_RQ(my_qp))) {
