Re: [openib-general] Timeline of IPoIB performance

2005-10-12 Thread Herbert Xu
On Wed, Oct 12, 2005 at 06:24:32PM -0700, Matt Leininger wrote: > > Is it still worth testing the TSO patches that Herbert suggested for > some of the 2.6.13-rc kernels? If you're still seeing a performance regression compared to 2.6.12-rc4, then yes (According to the figures in your message t

Re: [openib-general] Timeline of IPoIB performance

2005-10-12 Thread Matt Leininger
On Wed, 2005-10-12 at 11:28 -0700, Matt Leininger wrote: > On Wed, 2005-10-12 at 09:53 -0700, Roland Dreier wrote: > > Herbert> Try reverting the changeset > > > > Herbert> 314324121f9b94b2ca657a494cf2b9cb0e4a28cc > > > > Herbert> which lies between these two points and may be relevan

Re: [openib-general] Timeline of IPoIB performance

2005-10-12 Thread Matt Leininger
On Wed, 2005-10-12 at 09:53 -0700, Roland Dreier wrote: > Herbert> Try reverting the changeset > > Herbert> 314324121f9b94b2ca657a494cf2b9cb0e4a28cc > > Herbert> which lies between these two points and may be relevant. > > Matt, I pulled this out of git for you. I guess Herbert is s

Re: [openib-general] Timeline of IPoIB performance

2005-10-12 Thread Roland Dreier
Herbert> Try reverting the changeset Herbert> 314324121f9b94b2ca657a494cf2b9cb0e4a28cc Herbert> which lies between these two points and may be relevant. Matt, I pulled this out of git for you. I guess Herbert is suggesting to patch -R the below against 2.6.12-rc5: diff --git a/net/

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Roland Dreier
Roland> I might get a chance to do it tonight... I'll post if I do. I'm giving it a shot but I just can't reproduce this well on my systems. I do see a pretty big regression between 2.6.12-rc4 and 2.6.14-rc2, but 2.6.12-rc5 looks OK on my systems. I reflashed to FW 4.7.0 (mem-ful) and built

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Andi Kleen
On Tuesday 11 October 2005 01:30, Grant Grundler wrote: > On Mon, Oct 10, 2005 at 02:26:52PM -0700, Grant Grundler wrote: > ... > > > If it's interleaving, every other cacheline will be "local". > > ISTR AMD64 was page-interleaved but then got confused by documents > describing "128-bit" 2-way inte

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Roland Dreier
Matt> Yes, I'm using mem-full HCAs. I could try reflashing the Matt> firmware for memfree if that's of interest. No, probably not. If I get a chance I'll do the opposite (flash mem-free -> mem-full, since my HCAs do have memory) and see if it makes my results stable. Matt> I wa

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Matt Leininger
On Mon, 2005-10-10 at 16:38 -0700, Roland Dreier wrote: > Matt> Pretty consistent. Here are a few runs with 2.6.12-rc5 > Matt> with reboots in between each run. I'm using netperf-2.3pl1. > > That's interesting. I'm guessing you're using mem-ful HCAs? Yes, I'm using mem-full HCAs.

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Roland Dreier
Matt> Pretty consistent. Here are a few runs with 2.6.12-rc5 Matt> with reboots in between each run. I'm using netperf-2.3pl1. That's interesting. I'm guessing you're using mem-ful HCAs? Given that your results are more stable than mine, if you're up for it, you could install git, cl

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Grant Grundler
On Mon, Oct 10, 2005 at 02:26:52PM -0700, Grant Grundler wrote: ... > If it's interleaving, every other cacheline will be "local". ISTR AMD64 was page-interleaved but then got confused by documents describing "128-bit" 2-way interleave. I now realize the 128bit is refering to interleave between tw

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Matt Leininger
On Mon, 2005-10-10 at 11:23 -0700, Roland Dreier wrote: > > 2.6.12-rc5 in-kernel1 405 < > > 2.6.12-rc4 in-kernel1 470 < > > I was optimistic when I saw this, because the changeover to git > occurred with 2.6.12-rc2, so I thought I could use git bisec

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Rick Jones
Roland Dreier wrote: Rick> Which rev of netperf are you using, and areyou using the Rick> "confidence intervals" options (-i, -I)? for a long time, Rick> the linux-unique behaviour of returning the overhead bytes Rick> for SO_[SND|RCV]BUF and them being 2X what one gives in R

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Grant Grundler
On Mon, Oct 10, 2005 at 11:23:45AM -0700, Roland Dreier wrote: > > 2.6.12-rc5 in-kernel1 405 < > > 2.6.12-rc4 in-kernel1 470 < > > I was optimistic when I saw this, because the changeover to git > occurred with 2.6.12-rc2, so I thought I could use gi

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Roland Dreier
Rick> Which rev of netperf are you using, and areyou using the Rick> "confidence intervals" options (-i, -I)? for a long time, Rick> the linux-unique behaviour of returning the overhead bytes Rick> for SO_[SND|RCV]BUF and them being 2X what one gives in Rick> setsockopt() gave

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Rick Jones
Roland Dreier wrote: > 2.6.12-rc5 in-kernel1 405 < > 2.6.12-rc4 in-kernel1 470 < I was optimistic when I saw this, because the changeover to git occurred with 2.6.12-rc2, so I thought I could use git bisect to track down exactly when the performanc

Re: [openib-general] Timeline of IPoIB performance

2005-10-10 Thread Roland Dreier
> 2.6.12-rc5 in-kernel1 405 < > 2.6.12-rc4 in-kernel1 470 < I was optimistic when I saw this, because the changeover to git occurred with 2.6.12-rc2, so I thought I could use git bisect to track down exactly when the performance regression happened.

Re: [openib-general] Timeline of IPoIB performance

2005-10-07 Thread Matt Leininger
I'm adding netdev to this thread to see if they can help. I'm seeing an IPoIB (IP over InfiniBand) netperf performance drop off, of up to 90 MB/s, when using kernels newer than 2.6.11. This doesn't appear to be an OpenIB IPoIB issue since the older in-kernel IB for 2.6.11 and a recent svn3687 sna

Re: [openib-general] Timeline of IPoIB performance

2005-10-07 Thread Matt Leininger
On Fri, 2005-10-07 at 18:16 -0700, Roland Dreier wrote: > I wonder if this BIC bug has anything to do with it: > http://lkml.org/lkml/2005/10/7/230 > I'm not sure this helps. I'm seeing the performance drop of happen between 2.6.12-rc4 (470 MB/s) and 2.6.12-rc5 (405 MB/s). I'll send out my

Re: [openib-general] Timeline of IPoIB performance

2005-10-07 Thread Roland Dreier
I wonder if this BIC bug has anything to do with it: http://lkml.org/lkml/2005/10/7/230 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/li

Re: [openib-general] Timeline of IPoIB performance

2005-10-07 Thread Roland Dreier
Hmm, looks like something in the network stack must have changed. > 2.6.12 in-kernel1 406 > 2.6.11 in-kernel1 464 This looks like the biggest dropoff. I can think of two things that would be interesting to do if you or anyone else has time. First, tak

Re: [openib-general] Timeline of IPoIB performance

2005-10-07 Thread Parks Fields
Matt, I have seen the same thing. I just didn't relate it to the Kernel. My IPoIB performance is down to ~340MB/sec with 2.6.12.1 and svn 3040. With 2.6.13 and svn 3490 the peak is 402MB/sec. At 02:06 AM 10/7/2005, Matt Leininger wrote: I'm seeing an IPoIB netperf performance drop off

Re: [openib-general] Timeline of IPoIB performance

2005-10-07 Thread Hal Rosenstock
Hi Matt, On Fri, 2005-10-07 at 04:06, Matt Leininger wrote: > I'm seeing an IPoIB netperf performance drop off, up to 90 MB/s, when > using kernels newer than 2.6.11. This doesn't appear to be an OpenIB > IPoIB issue since the in-kernel and a recent svn3687 snapshot both have > the same performan

[openib-general] Timeline of IPoIB performance

2005-10-07 Thread Matt Leininger
I'm seeing an IPoIB netperf performance drop off, up to 90 MB/s, when using kernels newer than 2.6.11. This doesn't appear to be an OpenIB IPoIB issue since the in-kernel and a recent svn3687 snapshot both have the same performance (464 MB/s) with 2.6.11. I used the same kernel config file as a s