Re: [PATCH 0/5] of_platform_driver and OF_DEVICE removal
On Monday 22 April 2013, Rob Herring wrote: From: Rob Herring rob.herr...@calxeda.com This series is a relatively straight-forward removal of the last remaining user of of_platform_driver (ibmebus) and removal of CONFIG_OF_DEVICE which is always enabled when CONFIG_OF is enabled. Compile tested on powerpc and sparc. Acked-by: Arnd Bergmann a...@arndb.de -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: NFS over RDMA benchmark
-Original Message-
From: Peng Tao [mailto:bergw...@gmail.com]
Sent: Friday, April 19, 2013 05:28
To: Yan Burman
Cc: J. Bruce Fields; Tom Tucker; linux-rdma@vger.kernel.org; linux-
n...@vger.kernel.org
Subject: Re: NFS over RDMA benchmark
On Wed, Apr 17, 2013 at 10:36 PM, Yan Burman y...@mellanox.com
wrote:
Hi.
I've been trying to do some benchmarks for NFS over RDMA and I seem to
only get about half of the bandwidth that the HW can give me.
My setup consists of 2 servers each with 16 cores, 32Gb of memory, and
Mellanox ConnectX3 QDR card over PCI-e gen3.
These servers are connected to a QDR IB switch. The backing storage on the
server is tmpfs mounted with noatime.
I am running kernel 3.5.7.
When running ib_send_bw, I get 4.3-4.5 GB/sec for block sizes 4-512K.
When I run fio over rdma mounted nfs, I get 260-2200MB/sec for the same
block sizes (4-512K). running over IPoIB-CM, I get 200-980MB/sec.
I got to these results after the following optimizations:
1. Setting IRQ affinity to the CPUs that are part of the NUMA node the
card is on 2. Increasing
/proc/sys/sunrpc/svc_rdma/max_outbound_read_requests and
/proc/sys/sunrpc/svc_rdma/max_requests to 256 on server 3. Increasing
RPCNFSDCOUNT to 32 on server
Did you try to affine nfsd to corresponding CPUs where your IB card locates?
Given that you see a bottleneck on CPU (as in your later email), it might be
worth trying.
I tried to affine nfsd to CPUs on the NUMA node the IB card is on.
I also set tmpfs memory policy to allocate from the same NUMA node.
I did not see big difference.
4. FIO arguments: --rw=randread --bs=4k --numjobs=2 --iodepth=128
--ioengine=libaio --size=10k --prioclass=1 --prio=0 --cpumask=255
--loops=25 --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1
--norandommap --group_reporting --exitall --buffered=0
On client side, it may be good to affine FIO processes and nfsiod to CPUs
where IB card locates as well, in case client is the bottleneck.
I am doing that - cpumask=255 affines it to the NUMA node my card is on.
For some reason doing taskset on nfsiod fails.
--
Thanks,
Tao
N�r��yb�X��ǧv�^�){.n�+{��ٚ�{ay� ʇڙ�,j��f���h���z� �w���
���j:+v���w�j�mzZ+�ݢj��!�i
Re: [PATCH V4 for-next 1/5] IB/core: Add RSS and TSS QP groups - suggesting BOF during OFA conf to further discuss that
On Mon, Apr 15, 2013 at 4:21 PM, Or Gerlitz or.gerl...@gmail.com wrote: Actually these comments and questions on the series come just a week before the annual OFA gathering, personally, I will not be there nor Shlomo who is the author of the patches, but Tzahi Oved from Mellanox who lead the architecture for the QP group concept is planned to attend and same for Sean, Roland and I hope you (Ira) too, same for Ali Ayoub and Liran Liss from Mellanox who are attending too, all in all, nice quorum to get into a room and do white boarding, open discussion, laughing, yelling and what ever needed to get a consensus. [...] Sean, Tzahi -- I understand now that there have been few talkings @ the OFA meeting re this patch set. So what's the way to move forward, any concrete feedback that needs to be addressed here? This series is hanging since May 2012 and I'd like to see it gets in for 3.10, now if indeed Sean is OK with the general framework, please suggest. Or. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2] Use autoreconf in autogen.sh
On Apr 19, 2013, at 8:19 PM, Hefty, Sean sean.he...@intel.com wrote: It may help if you identify the library this patch is against. :) 3rd time sending will be the charm... :-) -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] Ad IB_MTU_1500|9000 enums.
On 04/20/2013 07:29 PM, Hefty, Sean wrote: This seems reasonable, but still concerns me a bit. The original version was flat out wrong because you can't re-arrange any exposed enum like this without requiring that all user space apps be recompiled. This is especially true because ibv_mtu_enum_to_int is an inline ib_mtu_enum_to_int() is a kernel function, not user space, so I think we're fine here, unless you're concerned about drivers built out of tree. Well, although *I* might have to worry about out of kernel drivers, I wouldn't suggest such for upstream. However, for some reason I had it in my mind when I was reading the patch that it was against libibverbs. That's what I get for staying up late and reviewing when I'm tired :-/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] libiberbs: .gitignore updates and rename configure.in-.ac
Added some entries to config/.gitignore for newer versions of the GNU Autotools. Also renamed configure.in - configure.ac to accomodate newer GNU Autotools (http://lists.gnu.org/archive/html/autotools-announce/2012-11/msg0.html announced the intent to drop support for configure.in in future versions of Autoconf). Signed-off-by: Jeff Squyres jsquy...@cisco.com --- .gitignore | 6 + configure.ac | 74 configure.in | 74 3 files changed, 80 insertions(+), 74 deletions(-) create mode 100644 configure.ac delete mode 100644 configure.in diff --git a/.gitignore b/.gitignore index 78effef..d198dd1 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,7 @@ autom4te.cache aclocal.m4 stamp-h.in config.h.in +config.h.in~ config.log config.h .libs @@ -15,3 +16,8 @@ Makefile config.status stamp-h1 libtool +config/libtool.m4 +config/ltoptions.m4 +config/ltsugar.m4 +config/ltversion.m4 +config/lt~obsolete.m4 diff --git a/configure.ac b/configure.ac new file mode 100644 index 000..efdc5ac --- /dev/null +++ b/configure.ac @@ -0,0 +1,74 @@ +dnl Process this file with autoconf to produce a configure script. + +AC_PREREQ(2.57) +AC_INIT(libibverbs, 1.1.6, linux-rdma@vger.kernel.org) +AC_CONFIG_SRCDIR([src/ibverbs.h]) +AC_CONFIG_AUX_DIR(config) +AC_CONFIG_MACRO_DIR(config) +AC_CONFIG_HEADER(config.h) +AM_INIT_AUTOMAKE([foreign]) +m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) + +dnl Checks for programs +AC_PROG_CC +AC_GNU_SOURCE +AC_PROG_LN_S +AC_PROG_LIBTOOL + +LT_INIT + +AC_ARG_WITH([valgrind], +AC_HELP_STRING([--with-valgrind], +[Enable Valgrind annotations (small runtime overhead, default NO)])) +if test x$with_valgrind = x || test x$with_valgrind = xno; then +want_valgrind=no +AC_DEFINE([NVALGRIND], 1, [Define to 1 to disable Valgrind annotations.]) +else +want_valgrind=yes +if test -d $with_valgrind; then +CPPFLAGS=$CPPFLAGS -I$with_valgrind/include +fi +fi + +dnl Checks for libraries +AC_CHECK_LIB(dl, dlsym, [], +AC_MSG_ERROR([dlsym() not found. libibverbs requires libdl.])) +AC_CHECK_LIB(pthread, pthread_mutex_init, [], +AC_MSG_ERROR([pthread_mutex_init() not found. libibverbs requires libpthread.])) + +dnl Checks for header files. +AC_HEADER_STDC +AC_CHECK_HEADER(valgrind/memcheck.h, +[AC_DEFINE(HAVE_VALGRIND_MEMCHECK_H, 1, +[Define to 1 if you have the valgrind/memcheck.h header file.])], +[if test $want_valgrind = yes; then +AC_MSG_ERROR([Valgrind memcheck support requested, but valgrind/memcheck.h not found.]) +fi]) + +dnl Checks for typedefs, structures, and compiler characteristics. +AC_C_CONST + +AC_CACHE_CHECK(whether ld accepts --version-script, ac_cv_version_script, +[if test -n `$LD --help /dev/null 2/dev/null | grep version-script`; then + ac_cv_version_script=yes +else + ac_cv_version_script=no +fi]) + +if test $ac_cv_version_script = yes; then + LIBIBVERBS_VERSION_SCRIPT='-Wl,--version-script=$(srcdir)/src/libibverbs.map' +else +LIBIBVERBS_VERSION_SCRIPT= +fi +AC_SUBST(LIBIBVERBS_VERSION_SCRIPT) + +AC_CACHE_CHECK(for .symver assembler support, ac_cv_asm_symver_support, +[AC_TRY_COMPILE(, [asm(symbol:\n.symver symbol, api@ABI\n);], +ac_cv_asm_symver_support=yes, +ac_cv_asm_symver_support=no)]) +if test $ac_cv_asm_symver_support = yes; then +AC_DEFINE([HAVE_SYMVER_SUPPORT], 1, [assembler has .symver support]) +fi + +AC_CONFIG_FILES([Makefile libibverbs.spec]) +AC_OUTPUT diff --git a/configure.in b/configure.in deleted file mode 100644 index efdc5ac..000 --- a/configure.in +++ /dev/null @@ -1,74 +0,0 @@ -dnl Process this file with autoconf to produce a configure script. - -AC_PREREQ(2.57) -AC_INIT(libibverbs, 1.1.6, linux-rdma@vger.kernel.org) -AC_CONFIG_SRCDIR([src/ibverbs.h]) -AC_CONFIG_AUX_DIR(config) -AC_CONFIG_MACRO_DIR(config) -AC_CONFIG_HEADER(config.h) -AM_INIT_AUTOMAKE([foreign]) -m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) - -dnl Checks for programs -AC_PROG_CC -AC_GNU_SOURCE -AC_PROG_LN_S -AC_PROG_LIBTOOL - -LT_INIT - -AC_ARG_WITH([valgrind], -AC_HELP_STRING([--with-valgrind], -[Enable Valgrind annotations (small runtime overhead, default NO)])) -if test x$with_valgrind = x || test x$with_valgrind = xno; then -want_valgrind=no -AC_DEFINE([NVALGRIND], 1, [Define to 1 to disable Valgrind annotations.]) -else -want_valgrind=yes -if test -d $with_valgrind; then -CPPFLAGS=$CPPFLAGS -I$with_valgrind/include -fi -fi - -dnl Checks for libraries -AC_CHECK_LIB(dl, dlsym, [], -AC_MSG_ERROR([dlsym() not found. libibverbs requires libdl.])) -AC_CHECK_LIB(pthread, pthread_mutex_init, [], -AC_MSG_ERROR([pthread_mutex_init() not found. libibverbs requires libpthread.])) - -dnl Checks for header files. -AC_HEADER_STDC -AC_CHECK_HEADER(valgrind/memcheck.h, -
[PATCH 1/2] libibverbs: Use autoreconf in autogen.sh
The old sequence of Autotools commands listed in autogen.sh is no longer correct. Instead, just use the single autoreconf command, which will invoke all the Right Autotools commands in the correct order. Signed-off-by: Jeff Squyres jsquy...@cisco.com --- autogen.sh | 6 +- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/autogen.sh b/autogen.sh index fd47839..6c9233e 100755 --- a/autogen.sh +++ b/autogen.sh @@ -1,8 +1,4 @@ #! /bin/sh set -x -aclocal -I config -libtoolize --force --copy -autoheader -automake --foreign --add-missing --copy -autoconf +autoreconf -ifv -I config -- 1.8.1.1 -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] Ad IB_MTU_1500|9000 enums.
On Apr 22, 2013, at 1:30 PM, Doug Ledford dledf...@redhat.com wrote: However, for some reason I had it in my mind when I was reading the patch that it was against libibverbs. That's what I get for staying up late and reviewing when I'm tired :-/ There were other patches against libibverbs that were submitted at the same time. That being said, I see two obvious ways to go forward, both of which have pros/cons: 1. Extend the enum ib_mtu to include new enum values for 1500 and 9000 -- probably with a different prefix to indicate that they're not IBTA-sanctioned values (note that this will also require corresponding changes in libibverbs, since MTU values get passed up from kernel to userspace). PRO: fixes the immediate problem PRO: probably the lowest impact solution; just adding some more enum values CON: weird naming (IB_ and RDMA_ prefixes in the same ib_mtu enum; probably something similar in userspace) CON: doesn't do anything to address other MTU values (e.g., what if someone has an MTU of 1498?) 2. Change all instances of ib_mtu/ibv_mtu to an int. Code such as switch(mtu) case IBV_MTU_1024: ... will need to be updated to switch(mtu) case 1024: PRO: solves the problem for all MTU values PRO: eliminates the enum-to-int translation functions CON: much driver code will need to be updated per above, and also update logic checking for out-of-bounds MTU calues CON: similarly, userspace apps will need to be updated; it might be worthwhile to bump libibverbs to 2.x, and then intentionally change the MTU field names in ibv_port_attr and ibv_qp_attr so that apps using those fields will fail to compile with libibverbs 2.x (and therefore forcibly realize they need to adapt to the new int MTU values) -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] Ad IB_MTU_1500|9000 enums.
On 04/22/2013 02:00 PM, Jeff Squyres (jsquyres) wrote: 2. Change all instances of ib_mtu/ibv_mtu to an int. Code such as switch(mtu) case IBV_MTU_1024: ... will need to be updated to switch(mtu) case 1024: PRO: solves the problem for all MTU values PRO: eliminates the num-to-int translation functions CON: much driver code will need to be updated per above, and also update logic checking for out-of-bounds MTU calues CON: similarly, userspace apps will need to be updated; it might be worthwhile to bump libibverbs to 2.x, and then intentionally change the MTU field names in ibv_port_attr and ibv_qp_attr so that apps using those fields will fail to compile with libibverbs 2.x (and therefore forcibly realize they need to adapt to the new int MTU values) I was actually thinking that an ibverbs API version 2.0 might be an interesting way to go. The proliferation of non-IB link layers providing the verbs API make some of the original assumptions of IB link layer in the original API obsolete. But, if we were to do that, I'd take some time to really think the issue over and try to catch all of the needed updates in one go. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
