Re: [openib-general] Timeline of IPoIB performance
Herbert Try reverting the changeset
Herbert 314324121f9b94b2ca657a494cf2b9cb0e4a28cc
Herbert which lies between these two points and may be relevant.
Matt, I pulled this out of git for you. I guess Herbert is suggesting
to patch -R the below against 2.6.12-rc5:
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 79835a6..5bad504 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4355,16 +4355,7 @@ int tcp_rcv_established(struct sock *sk,
goto no_ack;
}
- if (eaten) {
- if (tcp_in_quickack_mode(tp)) {
- tcp_send_ack(sk);
- } else {
- tcp_send_delayed_ack(sk);
- }
- } else {
- __tcp_ack_snd_check(sk, 0);
- }
-
+ __tcp_ack_snd_check(sk, 0);
no_ack:
if (eaten)
__kfree_skb(skb);
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
On Wed, 2005-10-12 at 09:53 -0700, Roland Dreier wrote: Herbert Try reverting the changeset Herbert 314324121f9b94b2ca657a494cf2b9cb0e4a28cc Herbert which lies between these two points and may be relevant. Matt, I pulled this out of git for you. I guess Herbert is suggesting to patch -R the below against 2.6.12-rc5: I applied your patch suggest by Herbert: http://www.mail-archive.com/openib-general%40openib.org/msg11415.html to my 2.6.12-rc5 tree and IPoIB performance improved back to the ~475 MB/s range for my EM64T system. The data is below. I'm building/testing 2.6.14-rc4 with and without this patch now. All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0 dual EM64T 3.2 GHz PCIe IB HCA (memfull) Kernel OpenIBmsi_x netperf (MB/s) 2.6.14-rc3 in-kernel1 374 2.6.13.2svn3627 1 386 2.6.13.2in-kernel1 394 2.6.12.5-lustre in-kernel1 399 2.6.12.5in-kernel1 402 2.6.12 in-kernel1 406 2.6.12-rc6 in-kernel1 407 2.6.12-rc5 in-kernel1 405 2.6.12-rc5 - remove changeset 314324121f9b94b2ca657a494cf2b9cb0e4a28cc in-kernel1 474 2.6.12-rc4 in-kernel1 470 2.6.12-rc3 in-kernel1 466 2.6.12-rc2 in-kernel1 469 2.6.12-rc1 in-kernel1 466 2.6.11 in-kernel1 464 2.6.11 svn3687 1 464 2.6.9-11.ELsmp svn3513 1 425 (Woody's results, 3.6Ghz EM64T) - Matt ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
On Wed, 2005-10-12 at 11:28 -0700, Matt Leininger wrote: On Wed, 2005-10-12 at 09:53 -0700, Roland Dreier wrote: Herbert Try reverting the changeset Herbert 314324121f9b94b2ca657a494cf2b9cb0e4a28cc Herbert which lies between these two points and may be relevant. Matt, I pulled this out of git for you. I guess Herbert is suggesting to patch -R the below against 2.6.12-rc5: I applied your patch suggest by Herbert: http://www.mail-archive.com/openib-general%40openib.org/msg11415.html I backed out this patch out of a few other kernels and always see a performance improvement. This gets back ~50-60 MB/s of the 90-100 MB/s drop off in IPoIB performance. Is it still worth testing the TSO patches that Herbert suggested for some of the 2.6.13-rc kernels? Thanks, - Matt All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0 dual EM64T 3.2 GHz PCIe IB HCA (memfull) Kernel OpenIBmsi_x netperf (MB/s) 2.6.14-rc4 in-kernel1 434 (backed out patch) 2.6.14-rc4 in-kernel1 385 2.6.13.2svn3627 1 446 (backed out patch) 2.6.13.2svn3627 1 386 2.6.13.2in-kernel1 394 2.6.12.5in-kernel1 464 (backed out patch) 2.6.12.5in-kernel1 402 2.6.12-rc6 in-kernel1 470 (backed out patch) 2.6.12-rc6 in-kernel1 407 2.6.12-rc5 in-kernel1 474 (backed out patch) 2.6.12-rc5 in-kernel1 405 2.6.9-11.ELsmp svn3513 1 425 (Woody's results, 3.6Ghz EM64T) ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
On Wed, Oct 12, 2005 at 06:24:32PM -0700, Matt Leininger wrote:
Is it still worth testing the TSO patches that Herbert suggested for
some of the 2.6.13-rc kernels?
If you're still seeing a performance regression compared to
2.6.12-rc4, then yes (According to the figures in your message
there does seem to be a bit of loss after the release of 2.6.12).
The patch you reverted may degrade the performance on the receiver.
The TSO patches may be causing some degradation on your sender.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmVHI~} [EMAIL PROTECTED]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
2.6.12-rc5 in-kernel1 405 2.6.12-rc4 in-kernel1 470 I was optimistic when I saw this, because the changeover to git occurred with 2.6.12-rc2, so I thought I could use git bisect to track down exactly when the performance regression happened. However, I haven't been able to get numbers that are stable enough to track this down. I have two systems, both HP DL145s with dual Opteron 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the completion interrupt affinity set to CPU 0, and taskset 2 to run netserver and netperf on CPU 1. With default netperf parameters (just -H otherguy) I get numbers between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5. The numbers are quite consistent between reboots, but if I reboot the system (even keeping the kernel identical), I see large performance changes. Presumably something is happening like the cache coloring of some hot data structures changing semi-randomly depending on the timing of various initialations. Matt, how stable are your numbers? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
Roland Dreier wrote: 2.6.12-rc5 in-kernel1 405 2.6.12-rc4 in-kernel1 470 I was optimistic when I saw this, because the changeover to git occurred with 2.6.12-rc2, so I thought I could use git bisect to track down exactly when the performance regression happened. However, I haven't been able to get numbers that are stable enough to track this down. I have two systems, both HP DL145s with dual Opteron 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the completion interrupt affinity set to CPU 0, and taskset 2 to run netserver and netperf on CPU 1. With default netperf parameters (just -H otherguy) I get numbers between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5. The numbers are quite consistent between reboots, but if I reboot the system (even keeping the kernel identical), I see large performance changes. Presumably something is happening like the cache coloring of some hot data structures changing semi-randomly depending on the timing of various initialations. Which rev of netperf are you using, and areyou using the confidence intervals options (-i, -I)? for a long time, the linux-unique behaviour of returning the overhead bytes for SO_[SND|RCV]BUF and them being 2X what one gives in setsockopt() gave netperf some trouble - the socket buffer would double in size each iteration on a confidence interval run. Later netperf versions (late 2.3, and 2.4.X) have a kludge for this. Slightly related to that, IIRC, the linux receiver code adjusts the advertised window as the connection goes along - how far the receive code opens the window may change from run to run - might that have an effect? If there is a way to get the linux receiver to simply advertise the full window from the beginning that might help minimize the number of variables. Are there large changes in service demand along with the large performance changes? FWIW, on later netperfs the -T option should allow you to specify the CPU on which netperf and/or netserver run, although I've had some trouble reliably detecting the right sched_setaffinity syntax among the releases. rick jones ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
Rick Which rev of netperf are you using, and areyou using the Rick confidence intervals options (-i, -I)? for a long time, Rick the linux-unique behaviour of returning the overhead bytes Rick for SO_[SND|RCV]BUF and them being 2X what one gives in Rick setsockopt() gave netperf some trouble - the socket buffer Rick would double in size each iteration on a confidence interval Rick run. Later netperf versions (late 2.3, and 2.4.X) have a Rick kludge for this. I believe it's netperf 2.2. I'm not using any confidence interval stuff. However, the variation is not between single runs of netperf -- if I do 5 runs of netperf in a row, I get roughly the same number from each run. For example, I might see something like TCP STREAM TEST to 192.168.145.2 : histogram Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 87380 16384 1638410.003869.82 and then TCP STREAM TEST to 192.168.145.2 : histogram Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 87380 16384 1638410.003862.41 for two successive runs. However, if I reboot the system into the same kernel (ie everything set up exactly the same), the same invocation of netperf might give TCP STREAM TEST to 192.168.145.2 : histogram Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 87380 16384 1638410.004389.20 Rick Are there large changes in service demand along with the Rick large performance changes? Not sure. How do I have netperf report service demand? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
Roland Dreier wrote: Rick Which rev of netperf are you using, and areyou using the Rick confidence intervals options (-i, -I)? for a long time, Rick the linux-unique behaviour of returning the overhead bytes Rick for SO_[SND|RCV]BUF and them being 2X what one gives in Rick setsockopt() gave netperf some trouble - the socket buffer Rick would double in size each iteration on a confidence interval Rick run. Later netperf versions (late 2.3, and 2.4.X) have a Rick kludge for this. I believe it's netperf 2.2. That's rather old. I literally just put 2.4.1 out on ftp.cup.hp.com - probably better to use that if possible. Not that it will change the variability just that I like it when people are up-to-date on the versions :) If nothing else, the 2.4.X version(s) have a much improved (hopefully) manual in doc/ [If you are really maschochistic, the very first release of netperf 4.0.0 source has happened. I can make no guarantees as to its actually working at the moment though :) Netperf4 is going to be the stream for the multiple-connection, multiple system tests rather than the single-connection nature of netperf2] I'm not using any confidence interval stuff. However, the variation is not between single runs of netperf -- if I do 5 runs of netperf in a row, I get roughly the same number from each run. For example, I might see something like TCP STREAM TEST to 192.168.145.2 : histogram Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 87380 16384 1638410.003869.82 and then TCP STREAM TEST to 192.168.145.2 : histogram Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 87380 16384 1638410.003862.41 for two successive runs. However, if I reboot the system into the same kernel (ie everything set up exactly the same), the same invocation of netperf might give TCP STREAM TEST to 192.168.145.2 : histogram Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 87380 16384 1638410.004389.20 Rick Are there large changes in service demand along with the Rick large performance changes? Not sure. How do I have netperf report service demand? Ask for CPU utilization with -c (local) and -C (remote). The /proc/stat stuff used by Linux does not need calibration (IIRC) so you don't have to worry about that. If cache effects are involved, you can make netperf harder or easier on the caches by altering the size of the send and/or recv buffer rings. By default they are one more than the socket buffer size divided by the send size, but you can make them larger or smaller with the -W option. These days I use a 128K socket buffer and 32K send for the canonical (although not default :) netperf TCP_STREAM test: netperf -H remote -c -C -- -s 128K -S 128K -m 32K In netperf-speak K == 1024, k == 1000, M == 2^20, m == 10^6, G == 2^40, g == 10^9... rick jones ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
On Mon, 2005-10-10 at 11:23 -0700, Roland Dreier wrote: 2.6.12-rc5 in-kernel1 405 2.6.12-rc4 in-kernel1 470 I was optimistic when I saw this, because the changeover to git occurred with 2.6.12-rc2, so I thought I could use git bisect to track down exactly when the performance regression happened. However, I haven't been able to get numbers that are stable enough to track this down. I have two systems, both HP DL145s with dual Opteron 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the completion interrupt affinity set to CPU 0, and taskset 2 to run netserver and netperf on CPU 1. With default netperf parameters (just -H otherguy) I get numbers between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5. The numbers are quite consistent between reboots, but if I reboot the system (even keeping the kernel identical), I see large performance changes. Presumably something is happening like the cache coloring of some hot data structures changing semi-randomly depending on the timing of various initialations. Matt, how stable are your numbers? Pretty consistent. Here are a few runs with 2.6.12-rc5 with reboots in between each run. I'm using netperf-2.3pl1. Run 1: TCP STREAM TEST to 10.128.20.6 Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.KBytes /s % T % T us/KB us/KB 87380 16384 1638410.00 410302.39 99.8992.094.869 4.489 Run 2: (after another reboot) TCP STREAM TEST to 10.128.20.6 Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.KBytes /s % T % T us/KB us/KB 87380 16384 1638410.00 409510.33 99.8991.594.879 4.473 Run 3: (after reboot) TCP STREAM TEST to 10.128.20.6 Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.KBytes /s % T % T us/KB us/KB 87380 16384 1638410.00 404354.11 99.8991.394.941 4.520 I see the same variance in netperf results if I don't reboot between runs. - Matt ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
On Mon, Oct 10, 2005 at 02:26:52PM -0700, Grant Grundler wrote: ... If it's interleaving, every other cacheline will be local. ISTR AMD64 was page-interleaved but then got confused by documents describing 128-bit 2-way interleave. I now realize the 128bit is refering to interleave between two banks of memory behind each memory controller. ie 2 * 128-bit provides in the 32-byte cacheline size that most x86 programs expect. Anyway, I'm hoping that we'll see a consistent result if node interleave is turned off. sorry for the confusion, grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
Matt Pretty consistent. Here are a few runs with 2.6.12-rc5 Matt with reboots in between each run. I'm using netperf-2.3pl1. That's interesting. I'm guessing you're using mem-ful HCAs? Given that your results are more stable than mine, if you're up for it, you could install git, clone Linus's tree, and then do a git bisect between 2.6.12-rc4 and 2.6.12-rc5 to narrow down the regression to a single commit (if in fact that's possible). - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
Matt Yes, I'm using mem-full HCAs. I could try reflashing the Matt firmware for memfree if that's of interest. No, probably not. If I get a chance I'll do the opposite (flash mem-free - mem-full, since my HCAs do have memory) and see if it makes my results stable. Matt I was hoping someone else would do this. :) I'll start Matt working on it tomorrow if no one else gets to it. I might get a chance to do it tonight... I'll post if I do. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
On Tuesday 11 October 2005 01:30, Grant Grundler wrote: On Mon, Oct 10, 2005 at 02:26:52PM -0700, Grant Grundler wrote: ... If it's interleaving, every other cacheline will be local. ISTR AMD64 was page-interleaved but then got confused by documents describing 128-bit 2-way interleave. I now realize the 128bit is refering to interleave between two banks of memory behind each memory controller. ie 2 * 128-bit provides in the 32-byte cacheline size that most x86 programs expect. The cache line size on K7 and K8 is 64 bytes. Anyway, I'm hoping that we'll see a consistent result if node interleave is turned off. Yes usually a good idea. -Andi ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
Roland I might get a chance to do it tonight... I'll post if I do. I'm giving it a shot but I just can't reproduce this well on my systems. I do see a pretty big regression between 2.6.12-rc4 and 2.6.14-rc2, but 2.6.12-rc5 looks OK on my systems. I reflashed to FW 4.7.0 (mem-ful) and built netperf 2.4.1. With 2.6.12-rc4 I've seen runs as slow as: TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.145.2 (192.168.145.2) port 0 AF_INET Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv SendRecv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.MBytes /s % S % U us/KB us/KB 87380 16384 1638410.00 553.71 37.46-1.002.642 -1.000 and with 2.6.12-rc5 I've seen runs as fast as: TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.145.2 (192.168.145.2) port 0 AF_INET Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv SendRecv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.MBytes /s % S % U us/KB us/KB 87380 16384 1638410.00 581.82 39.58-1.002.657 -1.000 so not much difference there. With 2.6.14-rc2, the best of 10 runs was: TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.145.2 (192.168.145.2) port 0 AF_INET Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv SendRecv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.MBytes /s % S % U us/KB us/KB 87380 16384 1638410.01 497.00 39.71-1.003.121 -1.000 so we've definitely lost something there. Time to do some more bisecting... - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Timeline of IPoIB performance
I'm seeing an IPoIB netperf performance drop off, up to 90 MB/s, when using kernels newer than 2.6.11. This doesn't appear to be an OpenIB IPoIB issue since the in-kernel and a recent svn3687 snapshot both have the same performance (464 MB/s) with 2.6.11. I used the same kernel config file as a starting point for each of these kernel builds. Have there been any changes in Linux that would explain these results? All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0 dual EM64T 3.2 GHz PCIe IB HCA (memfull) Kernel OpenIBmsi_x netperf (MB/s) 2.6.14-rc3 in-kernel1 374 2.6.13.2svn3627 1 386 2.6.13.2in-kernel1 394 2.6.12 in-kernel1 406 2.6.11 in-kernel1 464 2.6.11 svn3687 1 464 2.6.9-11.ELsmp svn3513 1 425 (Woody's results, 3.6Ghz EM64T) Thanks, - Matt ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
Hi Matt, On Fri, 2005-10-07 at 04:06, Matt Leininger wrote: I'm seeing an IPoIB netperf performance drop off, up to 90 MB/s, when using kernels newer than 2.6.11. This doesn't appear to be an OpenIB IPoIB issue since the in-kernel and a recent svn3687 snapshot both have the same performance (464 MB/s) with 2.6.11. I used the same kernel config file as a starting point for each of these kernel builds. Have there been any changes in Linux that would explain these results? All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0 dual EM64T 3.2 GHz PCIe IB HCA (memfull) Kernel OpenIBmsi_x netperf (MB/s) 2.6.14-rc3 in-kernel1 374 2.6.13.2svn3627 1 386 2.6.13.2in-kernel1 394 2.6.12 in-kernel1 406 2.6.11 in-kernel1 464 2.6.11 svn3687 1 464 2.6.9-11.ELsmp svn3513 1 425 (Woody's results, 3.6Ghz EM64T) There was already the following thread on netdev that I found: TCP Network performance degade from 2.4.18 to 2.6.10 http://marc.theaimsgroup.com/?l=linux-netdevm=112792558832125w=2 I think you should (cross)post this to netdev. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
Matt, I have seen the same thing. I just didn't relate it to the Kernel. My IPoIB performance is down to ~340MB/sec with 2.6.12.1 and svn 3040. With 2.6.13 and svn 3490 the peak is 402MB/sec. At 02:06 AM 10/7/2005, Matt Leininger wrote: I'm seeing an IPoIB netperf performance drop off, up to 90 MB/s, when using kernels newer than 2.6.11. This doesn't appear to be an OpenIB IPoIB issue since the in-kernel and a recent svn3687 snapshot both have the same performance (464 MB/s) with 2.6.11. I used the same kernel config file as a starting point for each of these kernel builds. Have there been any changes in Linux that would explain these results? All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0 dual EM64T 3.2 GHz PCIe IB HCA (memfull) Kernel OpenIBmsi_x netperf (MB/s) 2.6.14-rc3 in-kernel1 374 2.6.13.2svn3627 1 386 2.6.13.2in-kernel1 394 2.6.12 in-kernel1 406 2.6.11 in-kernel1 464 2.6.11 svn3687 1 464 2.6.9-11.ELsmp svn3513 1 425 (Woody's results, 3.6Ghz EM64T) Thanks, - Matt ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
Hmm, looks like something in the network stack must have changed. 2.6.12 in-kernel1 406 2.6.11 in-kernel1 464 This looks like the biggest dropoff. I can think of two things that would be interesting to do if you or anyone else has time. First, taking profiles of netperf runs between these two kernels and comparing might be enlightening. Also, it would be useful to pin down when the regression happened, so running the same test with 2.6.12-rc1 through 2.6.12-rc6 would be a good thing. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
I wonder if this BIC bug has anything to do with it: http://lkml.org/lkml/2005/10/7/230 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Timeline of IPoIB performance
I'm adding netdev to this thread to see if they can help. I'm seeing an IPoIB (IP over InfiniBand) netperf performance drop off, of up to 90 MB/s, when using kernels newer than 2.6.11. This doesn't appear to be an OpenIB IPoIB issue since the older in-kernel IB for 2.6.11 and a recent svn3687 snapshot both have the same performance (464 MB/s) with 2.6.11. I used the same kernel config file as a starting point for each of these kernel builds. Have there been any changes in Linux that would explain these results? Here is the hardware setup and netperf results using 'netperf -f -M -c -C -H IPoIB_ADDRESS All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0 dual EM64T 3.2 GHz PCIe IB HCA (memfull) Kernel OpenIBmsi_x netperf (MB/s) 2.6.14-rc3 in-kernel1 374 2.6.13.2svn3627 1 386 2.6.13.2in-kernel1 394 2.6.12.5-lustre in-kernel1 399 2.6.12.5in-kernel1 402 2.6.12 in-kernel1 406 2.6.12-rc6 in-kernel1 407 2.6.12-rc5 in-kernel1 405 2.6.12-rc4 in-kernel1 470 2.6.12-rc3 in-kernel1 466 2.6.12-rc2 in-kernel1 469 2.6.12-rc1 in-kernel1 466 2.6.11 in-kernel1 464 2.6.11 svn3687 1 464 2.6.9-11.ELsmp svn3513 1 425 (Woody's results, 3.6Ghz EM64T) Thanks, - Matt ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
