Re: RDMA will be reverted

2006-07-25 Thread David Miller
From: Evgeniy Polyakov [EMAIL PROTECTED] Date: Tue, 25 Jul 2006 09:51:28 +0400 On Mon, Jul 24, 2006 at 03:06:13PM -0700, David Miller ([EMAIL PROTECTED]) wrote: Furthermore, the VJ netchannel gains can be partially obtained from generic stateless facilities that we are going to get

Re: RDMA will be reverted

2006-07-25 Thread Evgeniy Polyakov
On Mon, Jul 24, 2006 at 11:48:53PM -0700, David Miller ([EMAIL PROTECTED]) wrote: And if that CPU is very busy? Linux should somehow tell NIC that some CPUs are valid and some are not right now, but not in a second, so scheduler must be tightly bound with network internals. Yes, it is

Re: RDMA will be reverted

2006-07-25 Thread David Miller
From: Evgeniy Polyakov [EMAIL PROTECTED] Date: Tue, 25 Jul 2006 10:59:21 +0400 As a side completely unrelated to either my or others work note :) - I think it is a nanooptimisation - we get a bit of performance here, and lose those bit in other place. When bag is filled, there is no much

Re: RDMA will be reverted

2006-07-25 Thread Evgeniy Polyakov
On Tue, Jul 25, 2006 at 12:33:44AM -0700, David Miller ([EMAIL PROTECTED]) wrote: From: Evgeniy Polyakov [EMAIL PROTECTED] Date: Tue, 25 Jul 2006 10:59:21 +0400 As a side completely unrelated to either my or others work note :) - I think it is a nanooptimisation - we get a bit of

Re: RDMA will be reverted

2006-07-25 Thread Rick Jones
David Miller wrote: From: Rick Jones [EMAIL PROTECTED] Date: Mon, 24 Jul 2006 17:55:24 -0700 Even enough bits for 1024 or 2048 CPUs in the single system image? I have seen 1024 touted by SGI, and with things going so multi-core, perhaps 16384 while sounding initially bizzare would be in the

Re: RDMA will be reverted

2006-07-25 Thread Andi Kleen
It may be a hardware interpretation, but doesn't it have non-trivial system implications - where one runs threads/processes etc? Only if you do process context RX processing. If you chose not to it doesn't have much influence. -Andi - To unsubscribe from this list: send the line unsubscribe

Re: RDMA will be reverted

2006-07-25 Thread Tom Tucker
On Mon, 2006-07-24 at 15:23 -0700, David Miller wrote: From: Tom Tucker [EMAIL PROTECTED] Date: Wed, 05 Jul 2006 12:09:42 -0500 A TOE net stack is closed source firmware. Linux engineers have no way to fix security issues that arise. As a result, only non-TOE users will receive security

Re: RDMA will be reverted

2006-07-24 Thread David Miller
From: Roland Dreier [EMAIL PROTECTED] Date: Tue, 04 Jul 2006 13:34:27 -0700 Well, here's a quick overview, leaving out some of the details. The difference between TOE and iWARP/RDMA is really the interface that they present. Thanks for the description Roland. It helps me understand the

Re: RDMA will be reverted

2006-07-24 Thread David Miller
From: Tom Tucker [EMAIL PROTECTED] Date: Wed, 05 Jul 2006 12:09:42 -0500 A TOE net stack is closed source firmware. Linux engineers have no way to fix security issues that arise. As a result, only non-TOE users will receive security updates, leaving random windows of vulnerability for each

Re: RDMA will be reverted

2006-07-24 Thread David Miller
From: Steve Wise [EMAIL PROTECTED] Date: Wed, 05 Jul 2006 12:50:34 -0500 However, iWARP devices _could_ integrate with netfilter. For most devices the connection request event (SYN) gets passed up to the host driver. So the driver can enforce filter rules then. This doesn't work. In order

RE: RDMA will be reverted

2006-07-24 Thread Caitlin Bestler
[EMAIL PROTECTED] wrote: From: Steve Wise [EMAIL PROTECTED] Date: Wed, 05 Jul 2006 12:50:34 -0500 However, iWARP devices _could_ integrate with netfilter. For most devices the connection request event (SYN) gets passed up to the host driver. So the driver can enforce filter rules then.

RE: RDMA will be reverted

2006-07-24 Thread Caitlin Bestler
[EMAIL PROTECTED] wrote: From: Tom Tucker [EMAIL PROTECTED] Date: Wed, 05 Jul 2006 12:09:42 -0500 A TOE net stack is closed source firmware. Linux engineers have no way to fix security issues that arise. As a result, only non-TOE users will receive security updates, leaving random windows

Re: RDMA will be reverted

2006-07-24 Thread Andi Kleen
For example, my idea to allow ESTABLISHED TCP socket demux to be done before netfilter is flawed. Connection tracking and NAT can change the packet ID and loop it back to us to hit exactly an ESTABLISHED TCP socket, therefore we must always hit netfilter first. Hmm, how does this happen? I

Re: RDMA will be reverted

2006-07-24 Thread Andi Kleen
On Tuesday 25 July 2006 01:22, David Miller wrote: From: Andi Kleen [EMAIL PROTECTED] Date: Tue, 25 Jul 2006 01:10:25 +0200 All the original costs of route, netfilter, TCP socket lookup all reappear as we make VJ netchannels fit all the rules of real practical systems, eliminating

Re: RDMA will be reverted

2006-07-24 Thread Rick Jones
This all sounds like the discussions we had within HP-UX between 10.20 and 11.0 concerning Inbound Packet Scheduling vs Thread Optimized Packet Scheduling. IPS was done by the 10.20 stack at the handoff between the driver and netisr. If the packet was not an IP datagram fragment, parts of the

Re: RDMA will be reverted

2006-07-24 Thread David Miller
From: Rick Jones [EMAIL PROTECTED] Date: Mon, 24 Jul 2006 17:29:05 -0700 Nirvana I suppose would be the addition of a field in the header which could be used for the determination of where to process. A Transport Protocol option I suppose, maybe the IPv6 flow id, but knuth only knows if

Re: RDMA will be reverted

2006-07-24 Thread Rick Jones
David Miller wrote: From: Rick Jones [EMAIL PROTECTED] Date: Mon, 24 Jul 2006 17:29:05 -0700 Nirvana I suppose would be the addition of a field in the header which could be used for the determination of where to process. A Transport Protocol option I suppose, maybe the IPv6 flow id, but knuth

Re: RDMA will be reverted

2006-07-24 Thread Rick Jones
It would have to be done in such a way as to not make the PAWS tests fail by accident. But I think it's doable. CPU ID and higher-order generation number such that whenever the process migrates to a lower-numbered CPU, the generation number is bumped to make the timestamp larger than before?

Re: RDMA will be reverted

2006-07-24 Thread Andi Kleen
Even enough bits for 1024 or 2048 CPUs in the single system image? MSI-X supports 255 sub interrupts max, most hardware probably much less (e.g. 8 seems to be a popular number). It can be always hashed to the existing CPUs. It's a nice idea but I think standard hashing + processing in

Re: RDMA will be reverted

2006-07-24 Thread David Miller
From: Rick Jones [EMAIL PROTECTED] Date: Mon, 24 Jul 2006 17:55:24 -0700 Even enough bits for 1024 or 2048 CPUs in the single system image? I have seen 1024 touted by SGI, and with things going so multi-core, perhaps 16384 while sounding initially bizzare would be in the realm of

Re: RDMA will be reverted

2006-07-24 Thread Andi Kleen
On Tuesday 25 July 2006 02:29, Rick Jones wrote: This all sounds like the discussions we had within HP-UX between 10.20 and 11.0 concerning Inbound Packet Scheduling vs Thread Optimized Packet Scheduling. We've also talking about this for many years, just no code so far. Or rather Linux so

Re: RDMA will be reverted

2006-07-24 Thread Evgeniy Polyakov
On Mon, Jul 24, 2006 at 03:06:13PM -0700, David Miller ([EMAIL PROTECTED]) wrote: Don't get too excited about VJ netchannels, more and more roadblocks to their practicality are being found every day. For example, my idea to allow ESTABLISHED TCP socket demux to be done before netfilter is

Re: What is RDMA (was: RDMA will be reverted)

2006-07-11 Thread Herbert Xu
On Fri, Jul 07, 2006 at 01:25:44PM -0500, Steve Wise wrote: Some IP networking is involved for this. IP addresses and port numbers are used by the RDMA Connection Manager. The motivation for this was two-fold, I think: 1) to simplify the connection setup model. The IB CM model was very

Re: RDMA will be reverted

2006-07-07 Thread David Miller
From: Tom Tucker [EMAIL PROTECTED] Date: Thu, 06 Jul 2006 00:25:03 -0500 This patch is about dotting I's and crossing T's, it's not about foundations. You assume that I've flat out rejected RDMA, in fact I haven't. I really don't have enough information to form a final opinion yet. There's

What is RDMA (was: RDMA will be reverted)

2006-07-07 Thread Herbert Xu
On Fri, Jul 07, 2006 at 06:53:20AM +, David Miller wrote: What I am saying, however, is that we need to understand the technology and the hooks you guys want before we put any of it in. Yes indeed. Here is what I've understood so far so let's see if we can start building a censensus. 1)

Re: RDMA will be reverted

2006-07-07 Thread Tom Tucker
On Thu, 2006-07-06 at 23:53 -0700, David Miller wrote: From: Tom Tucker [EMAIL PROTECTED] Date: Thu, 06 Jul 2006 00:25:03 -0500 This patch is about dotting I's and crossing T's, it's not about foundations. You assume that I've flat out rejected RDMA, in fact I haven't. I really don't

Re: What is RDMA (was: RDMA will be reverted)

2006-07-07 Thread Steve Wise
Great summation. Comments in-line... On Fri, 2006-07-07 at 18:11 +1000, Herbert Xu wrote: On Fri, Jul 07, 2006 at 06:53:20AM +, David Miller wrote: What I am saying, however, is that we need to understand the technology and the hooks you guys want before we put any of it in.

RE: RDMA will be reverted

2006-07-06 Thread Caitlin Bestler
Andi Kleen wrote: We're focusing on netfilter here. Is breaking netfilter really the only issue with this stuff? Another concern is that it will just not be able to keep up with a high rate of new connections or a high number of them (because the hardware has too limited state) Neither

Re: RDMA will be reverted

2006-07-06 Thread Herbert Xu
Tom Tucker [EMAIL PROTECTED] wrote: All that said, the proposed patch helps not only iWARP, but other transports (iSCSI, IB) as well. It is not large, invasive, Care to explain on how it helps those other technologies? Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu

Re: RDMA will be reverted

2006-07-06 Thread Tom Tucker
On Fri, 2006-07-07 at 00:08 +1000, Herbert Xu wrote: Tom Tucker [EMAIL PROTECTED] wrote: All that said, the proposed patch helps not only iWARP, but other transports (iSCSI, IB) as well. It is not large, invasive, Care to explain on how it helps those other technologies? The RDMA CMA

Re: RDMA will be reverted

2006-07-06 Thread Herbert Xu
On Thu, Jul 06, 2006 at 12:36:24PM -0500, Tom Tucker wrote: The RDMA CMA uses IP addresses and port numbers to create a uniform addressing scheme across all transport types. For IB, it is necessary to resolve IP addresses to IB GIDs. The ARP protocol is used to do this and a netfilter rule

Re: RDMA will be reverted

2006-07-06 Thread Tom Tucker
On Fri, 2006-07-07 at 10:03 +1000, Herbert Xu wrote: On Thu, Jul 06, 2006 at 12:36:24PM -0500, Tom Tucker wrote: The RDMA CMA uses IP addresses and port numbers to create a uniform addressing scheme across all transport types. For IB, it is necessary to resolve IP addresses to IB GIDs.

Re: RDMA will be reverted

2006-07-05 Thread Tom Tucker
On Sat, 2006-07-01 at 16:26 +0200, Andi Kleen wrote: On Saturday 01 July 2006 01:01, Tom Tucker wrote: On Fri, 2006-06-30 at 14:16 -0700, David Miller wrote: The TOE folks have tried to submit their hooks and drivers on several occaisions, and we've rejected it every time. iWARP

Re: RDMA will be reverted

2006-07-05 Thread Steve Wise
On Wed, 2006-07-05 at 12:09 -0500, Tom Tucker wrote: On Sat, 2006-07-01 at 16:26 +0200, Andi Kleen wrote: On Saturday 01 July 2006 01:01, Tom Tucker wrote: On Fri, 2006-06-30 at 14:16 -0700, David Miller wrote: The TOE folks have tried to submit their hooks and drivers on several

Re: RDMA will be reverted

2006-07-05 Thread David Miller
From: Roland Dreier [EMAIL PROTECTED] Date: Tue, 04 Jul 2006 13:34:30 -0700 Well, in the past it's seemed more like patches have been rejected not because of a blanket refusal to consider support for certain hardware, Then why in the world would we put up explicit web pages that say TOE is

Re: RDMA will be reverted

2006-07-05 Thread Roland Dreier
Then why in the world would we put up explicit web pages that say TOE is bad, here's a list of reasons why if we had any intention of ever adding support for these kinds of devices? I think there's a little bit of leap of logic there. Everyone agrees that winmodems are bad and yet there's

Re: RDMA will be reverted

2006-07-05 Thread David Miller
From: Roland Dreier [EMAIL PROTECTED] Date: Wed, 05 Jul 2006 13:29:35 -0700 The way forward seems to be to merge basic iWARP support that lives in drivers/infiniband, and then you can accept or reject things for better integration, like notifiers for routing changes. sarcasm Let's merge in

Re: RDMA will be reverted

2006-07-05 Thread Tom Tucker
On Wed, 2006-07-05 at 20:03 -0700, David Miller wrote: From: Roland Dreier [EMAIL PROTECTED] Date: Wed, 05 Jul 2006 13:29:35 -0700 The way forward seems to be to merge basic iWARP support that lives in drivers/infiniband, and then you can accept or reject things for better integration,

Re: RDMA will be reverted

2006-07-04 Thread Andy Gay
On Sat, 2006-07-01 at 16:26 +0200, Andi Kleen wrote: On Saturday 01 July 2006 01:01, Tom Tucker wrote: On Fri, 2006-06-30 at 14:16 -0700, David Miller wrote: The TOE folks have tried to submit their hooks and drivers on several occaisions, and we've rejected it every time. iWARP

Re: RDMA will be reverted

2006-07-04 Thread Roland Dreier
Andi Perhaps a good start of that discussion David asked for Andi would be if you could give us an overview of the differences Andi and how you avoid the TOE problems. Well, here's a quick overview, leaving out some of the details. The difference between TOE and iWARP/RDMA is really

Re: RDMA will be reverted

2006-07-04 Thread Roland Dreier
Roland stated that it has never been the case that we have rejected adding support for a certain class of devices on the kinds of merits being discussed in this thread. And I'm saying that TOE is such a case where we have emphatically done so. Well, in the past it's seemed more like

Re: RDMA will be reverted

2006-07-04 Thread Andi Kleen
So perhaps there's a good argument to make that a Linux system with the right hardware could be considered a core device. Likely any place you have such a system it would be dedicated to just moving data as well as possible, and let other systems do the other stuff. You wouldn't want your

Re: RDMA will be reverted

2006-07-04 Thread Andy Gay
On Tue, 2006-07-04 at 22:47 +0200, Andi Kleen wrote: So perhaps there's a good argument to make that a Linux system with the right hardware could be considered a core device. Likely any place you have such a system it would be dedicated to just moving data as well as possible, and let

Re: RDMA will be reverted

2006-07-04 Thread Andi Kleen
My point wasn't really about performance here, more that systems needing this level of performance (server farm is just an example) will probably be on an 'inside' network with firewalling being done elsewhere (at the access layer, to use the Cisco paradigm). It's just not good design to

Re: RDMA will be reverted

2006-07-04 Thread Andy Gay
On Wed, 2006-07-05 at 01:01 +0200, Andi Kleen wrote: My point wasn't really about performance here, more that systems needing this level of performance (server farm is just an example) will probably be on an 'inside' network with firewalling being done elsewhere (at the access layer, to

Re: RDMA will be reverted

2006-07-04 Thread Andi Kleen
Think of the ordinary single linux box somewhere at a rackspace provider which represents the majority of Linux boxes around. How many of those need 10G nics? Most of them already have gigabit. At some point they will have 10G too. Admittedly the iThingy under discussion here seems to

Re: RDMA will be reverted

2006-07-01 Thread Andi Kleen
On Saturday 01 July 2006 01:01, Tom Tucker wrote: On Fri, 2006-06-30 at 14:16 -0700, David Miller wrote: The TOE folks have tried to submit their hooks and drivers on several occaisions, and we've rejected it every time. iWARP != TOE Perhaps a good start of that discussion David asked

Re: RDMA will be reverted

2006-07-01 Thread David Miller
From: Tom Tucker [EMAIL PROTECTED] Date: Fri, 30 Jun 2006 18:01:43 -0500 On Fri, 2006-06-30 at 14:16 -0700, David Miller wrote: The TOE folks have tried to submit their hooks and drivers on several occaisions, and we've rejected it every time. iWARP != TOE You are taking my comment out

Re: RDMA will be reverted

2006-06-30 Thread Roland Dreier
You snipped my question about what specifically you wanted reverted, so I'm going to assume that after cooling down and understanding the situation, you're OK with everything that's in Linus's tree... The integration of iWARP with the Linux networking, while much better than TOE, is still

Re: RDMA will be reverted

2006-06-30 Thread David Miller
From: Roland Dreier [EMAIL PROTECTED] Date: Fri, 30 Jun 2006 13:51:19 -0700 And I can't think of a single other example of a case where we refused to merge a driver, not because of any issues with the driver code, but because we don't like the hardware it drives and think that people

Re: RDMA will be reverted

2006-06-30 Thread Tom Tucker
On Fri, 2006-06-30 at 14:16 -0700, David Miller wrote: The TOE folks have tried to submit their hooks and drivers on several occaisions, and we've rejected it every time. iWARP != TOE - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL

Re: RDMA will be reverted

2006-06-29 Thread Roland Dreier
David Roland, there is no way in the world we would have let David support for RDMA into the kernel tree had we seen and David reviewed it on netdev. I've discussed this with Andrew David Morton, and we'd like you to please revert all of the RDMA David code from Linus's tree

Re: RDMA will be reverted

2006-06-29 Thread Roland Dreier
Dave, you're going to have to be more specific. What do you mean by RDMA? The whole drivers/infiniband infrastructure, which handles RDMA over IB, has been upstream for a year and a half, and was in fact originally merged by you, so I'm guessing that's not what you mean. NET_DMA

Re: RDMA will be reverted

2006-06-29 Thread YOSHIFUJI Hideaki / 吉藤英明
Hello. In article [EMAIL PROTECTED] (at Thu, 29 Jun 2006 09:54:37 -0700), Roland Dreier [EMAIL PROTECTED] says: David Roland, there is no way in the world we would have let David support for RDMA into the kernel tree had we seen and David reviewed it on netdev. I've discussed

Re: RDMA will be reverted

2006-06-29 Thread YOSHIFUJI Hideaki / 吉藤英明
In article [EMAIL PROTECTED] (at Thu, 29 Jun 2006 10:35:56 -0700), Roland Dreier [EMAIL PROTECTED] says: Dave, you're going to have to be more specific. What do you mean by RDMA? The whole drivers/infiniband infrastructure, which handles RDMA over IB, has been upstream for a year

Re: RDMA will be reverted

2006-06-29 Thread Tom Tucker
On Thu, 2006-06-29 at 12:46 -0700, David Miller wrote: From: Roland Dreier [EMAIL PROTECTED] Date: Thu, 29 Jun 2006 09:54:37 -0700 In any case I think we need to find a way for Linux to support iWARP hardware, since there are users that want this, and (some of) the vendors are working

Re: RDMA will be reverted

2006-06-29 Thread Tom Tucker
[...snip...] community at large, Linux and Andrew, you? I'm not trying to be Linus sorry... spell checker... - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at

Re: RDMA will be reverted

2006-06-29 Thread David Miller
From: Tom Tucker [EMAIL PROTECTED] Date: Thu, 29 Jun 2006 15:11:06 -0500 Doesn't this position force vendors to build deeper adapters, not shallower adapters? Isn't this exactly the opposite of what is intended? Nope. Look at what the networking developers give a lot of attention and effort

Re: RDMA will be reverted

2006-06-29 Thread James Morris
On Thu, 29 Jun 2006, Tom Tucker wrote: On Thu, 2006-06-29 at 12:46 -0700, David Miller wrote: From: Roland Dreier [EMAIL PROTECTED] Date: Thu, 29 Jun 2006 09:54:37 -0700 In any case I think we need to find a way for Linux to support iWARP hardware, since there are users that want

Re: RDMA will be reverted

2006-06-29 Thread Tom Tucker
On Thu, 2006-06-29 at 13:19 -0700, David Miller wrote: From: Tom Tucker [EMAIL PROTECTED] Date: Thu, 29 Jun 2006 15:11:06 -0500 Doesn't this position force vendors to build deeper adapters, not shallower adapters? Isn't this exactly the opposite of what is intended? Nope. Look at

Re: RDMA will be reverted

2006-06-29 Thread David Miller
From: Tom Tucker [EMAIL PROTECTED] Date: Thu, 29 Jun 2006 15:47:13 -0500 I concede that features have been lost, but some applications benefit greatly from RDMA. For these applications and these customers, TOE folks give the same story... it's a broken record, really. Let us know when you can

Re: RDMA will be reverted

2006-06-29 Thread Andi Kleen
They do not bypass netfilter, they do not bypass packet scheduling, and yet they provide a hardware assist performance improvement for receive. Not that I'm a TOE advocate, but as long as the adapter leaves SYN/SYN-ACK to the stack and only turns on RDMA in ESTABLISHED it could at least do

Re: RDMA will be reverted

2006-06-29 Thread Tom Tucker
On Thu, 2006-06-29 at 13:53 -0700, David Miller wrote: From: Tom Tucker [EMAIL PROTECTED] Date: Thu, 29 Jun 2006 15:47:13 -0500 I concede that features have been lost, but some applications benefit greatly from RDMA. For these applications and these customers, TOE folks give the same

Re: RDMA will be reverted

2006-06-28 Thread Evgeniy Polyakov
On Wed, Jun 28, 2006 at 12:07:15AM -0700, David Miller ([EMAIL PROTECTED]) wrote: Roland, there is no way in the world we would have let support for RDMA into the kernel tree had we seen and reviewed it on netdev. I've discussed this with Andrew Morton, and we'd like you to please revert all

Re: RDMA will be reverted

2006-06-28 Thread Tom Tucker
On Wed, 2006-06-28 at 00:07 -0700, David Miller wrote: Roland, there is no way in the world we would have let support for RDMA into the kernel tree had we seen and reviewed it on netdev. I've discussed this with Andrew Morton, and we'd like you to please revert all of the RDMA code from

Re: RDMA will be reverted

2006-06-28 Thread Steve Wise
On Wed, 2006-06-28 at 00:07 -0700, David Miller wrote: Roland, there is no way in the world we would have let support for RDMA into the kernel tree had we seen and reviewed it on netdev. I've discussed this with Andrew Morton, and we'd like you to please revert all of the RDMA code from