Re: [PATCH RFC] iw_cxgb3: Support "iwarp-only" interfaces to avoid 4-tuple conflicts with the host stack.
Steve Wise wrote: Roland Dreier wrote: > > What's wrong with my suggestion of having the iwarp driver create an > > "iwX" interface to go with the normal "ethX" interface? It seems > > simpler to me, and there's a somewhat similar precedent with how > > mac80211 devices create both wlan0 and wmaster0 interfaces. > > - R. > > It seemed much more painful for me to implement. :-) > > I'll look into this, but I think for this to be done, the changes must > be in the cxgb3 driver, not the rdma driver, because the guts of the > netdev struct are all private to cxgb3. Remember that this interface > needs to still do non TCP traffic (like ARP and UDP)... > > Maybe you have something in mind here that I'm not thinking about? No, I was just spouting off. At least someone is looking at my patch. ;-) But the whole "create a magic alias" seems kind of unfriendly to the user. Maybe as you said, the cxgb3 net driver could create the alias for the iw_cxgb3 driver? I agree that it is not very user friendly. My current patch just utilizes the IP address alias logic in the IP stack. So when you do 'ifconfig ethxx:blah ipaddr up' it creates a struct in_ifaddr which contains a ptr to the real struct net_device that services this alias. However, from what I can tell, I cannot just create one of these without binding an address. So the driver cannot create the alias interface until it knows the ipaddr/netmask/etc. IE: if you say 'ifconfig ethxx:blah up' it fails... You must supply an address to get one of these created. To have the cxgb3 driver create something like 'iw0', I think it would need to create a full net_device struct. This makes the change much more complex. But perhaps its the right thing to do... Steve. Also, I could defer registering the device with the rdma core until the alias interface is created by the user. Thus the T3 device wouldn't be available for use until the ethxx:iw interface is created. And I could log a WARN or INFO message if the iw_cxgb3 module is loaded and no ethxx:iw alias exists. This would help clue in the user... Steve. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC] iw_cxgb3: Support "iwarp-only" interfaces to avoid 4-tuple conflicts with the host stack.
Roland Dreier wrote: > > What's wrong with my suggestion of having the iwarp driver create an > > "iwX" interface to go with the normal "ethX" interface? It seems > > simpler to me, and there's a somewhat similar precedent with how > > mac80211 devices create both wlan0 and wmaster0 interfaces. > > - R. > > It seemed much more painful for me to implement. :-) > > I'll look into this, but I think for this to be done, the changes must > be in the cxgb3 driver, not the rdma driver, because the guts of the > netdev struct are all private to cxgb3. Remember that this interface > needs to still do non TCP traffic (like ARP and UDP)... > > Maybe you have something in mind here that I'm not thinking about? No, I was just spouting off. At least someone is looking at my patch. ;-) But the whole "create a magic alias" seems kind of unfriendly to the user. Maybe as you said, the cxgb3 net driver could create the alias for the iw_cxgb3 driver? I agree that it is not very user friendly. My current patch just utilizes the IP address alias logic in the IP stack. So when you do 'ifconfig ethxx:blah ipaddr up' it creates a struct in_ifaddr which contains a ptr to the real struct net_device that services this alias. However, from what I can tell, I cannot just create one of these without binding an address. So the driver cannot create the alias interface until it knows the ipaddr/netmask/etc. IE: if you say 'ifconfig ethxx:blah up' it fails... You must supply an address to get one of these created. To have the cxgb3 driver create something like 'iw0', I think it would need to create a full net_device struct. This makes the change much more complex. But perhaps its the right thing to do... Steve. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC] iw_cxgb3: Support "iwarp-only" interfaces to avoid 4-tuple conflicts with the host stack.
> > What's wrong with my suggestion of having the iwarp driver create an > > "iwX" interface to go with the normal "ethX" interface? It seems > > simpler to me, and there's a somewhat similar precedent with how > > mac80211 devices create both wlan0 and wmaster0 interfaces. > > - R. > > It seemed much more painful for me to implement. :-) > > I'll look into this, but I think for this to be done, the changes must > be in the cxgb3 driver, not the rdma driver, because the guts of the > netdev struct are all private to cxgb3. Remember that this interface > needs to still do non TCP traffic (like ARP and UDP)... > > Maybe you have something in mind here that I'm not thinking about? No, I was just spouting off. But the whole "create a magic alias" seems kind of unfriendly to the user. Maybe as you said, the cxgb3 net driver could create the alias for the iw_cxgb3 driver? - R. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC] iw_cxgb3: Support "iwarp-only" interfaces to avoid 4-tuple conflicts with the host stack.
Roland Dreier wrote: > The sysadmin creates "for iwarp use only" alias interfaces of the form > "devname:iw*" where devname is the native interface name (eg eth0) for the > iwarp netdev device. The alias label can be anything starting with "iw". > The "iw" immediately after the ':' is the key used by the iwarp driver. What's wrong with my suggestion of having the iwarp driver create an "iwX" interface to go with the normal "ethX" interface? It seems simpler to me, and there's a somewhat similar precedent with how mac80211 devices create both wlan0 and wmaster0 interfaces. - R. It seemed much more painful for me to implement. :-) I'll look into this, but I think for this to be done, the changes must be in the cxgb3 driver, not the rdma driver, because the guts of the netdev struct are all private to cxgb3. Remember that this interface needs to still do non TCP traffic (like ARP and UDP)... Maybe you have something in mind here that I'm not thinking about? Steve. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC] iw_cxgb3: Support "iwarp-only" interfaces to avoid 4-tuple conflicts with the host stack.
> The sysadmin creates "for iwarp use only" alias interfaces of the form > "devname:iw*" where devname is the native interface name (eg eth0) for the > iwarp netdev device. The alias label can be anything starting with "iw". > The "iw" immediately after the ':' is the key used by the iwarp driver. What's wrong with my suggestion of having the iwarp driver create an "iwX" interface to go with the normal "ethX" interface? It seems simpler to me, and there's a somewhat similar precedent with how mac80211 devices create both wlan0 and wmaster0 interfaces. - R. - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH RFC] iw_cxgb3: Support "iwarp-only" interfaces to avoid 4-tuple conflicts with the host stack.
Roland/All,
Here is the first swipe at keeping iwarp connections on their own ip
addresses to avoid conflicts with the host stack.
- this is a request for comments
- it is not yet tested fully (tested a prototype of the initial concept)
- still needs serialization/locking
- stays in our RDMA sandbox ;-)
For background reading (if you dare), see:
http://www.mail-archive.com/[EMAIL PROTECTED]/msg05162.html
and
http://www.mail-archive.com/netdev@vger.kernel.org/msg44312.html
Also: I'm on vacation starting tomorrow until Tuesday 9/4. I'll address
comments when I return...
Steve.
---
iw_cxgb3: Support "iwarp-only" interfaces to avoid 4-tuple conflicts with
the host stack.
Design:
The sysadmin creates "for iwarp use only" alias interfaces of the form
"devname:iw*" where devname is the native interface name (eg eth0) for the
iwarp netdev device. The alias label can be anything starting with "iw".
The "iw" immediately after the ':' is the key used by the iwarp driver.
EG:
ifconfig eth0 192.168.70.123 up
ifconfig eth0:iw1 192.168.71.123 up
ifconfig eth0:iw2 192.168.72.123 up
In the above example, 192.168.70/24 is for TCP traffic, while
192.168.71/24 and 192.168.72/24 are for iWARP/RDMA use.
The rdma-only interface must be on its own subnet. This allows routing
all rdma traffic onto this interface.
The iWARP driver must translate all listens on address 0.0.0.0 to the
set of rdma-only ip addresses. This prevents incoming connects to the
TCP ipaddresses from going up the rdma stack.
Implementation Details:
- The iwarp driver registers for inetaddr events via
register_inetaddr_notifier(). This allows tracking the iwarp-only
addresses/subnets as they get added and deleted. The iwarp driver
maintains a list of the current iwarp-only addresses.
- The iwarp driver builds the list of iwarp-only addresses for its devices
at module insert time. This is needed because the inetaddr notifier
callbacks don't "replay" address-add events when someone registers.
So the driver must build the initial list at module load time.
- When a listen is done on address 0.0.0.0, then the iwarp driver must
translate that into a set of listens on the iwarp-only addresses.
- When a new iwarp-only address is added or removed, the iwarp driver
must traverse the set of listening endpoints and update them accordingly.
This allows an application to bind to 0.0.0.0 prior to the iwarp-only
interfaces being configured. It also allows changing the iwarp-only set
of addresses and getting the expected behavior for apps already bound
to 0.0.0.0.
Signed-off-by: Steve Wise <[EMAIL PROTECTED]>
---
drivers/infiniband/hw/cxgb3/iwch.c| 116 +
drivers/infiniband/hw/cxgb3/iwch.h| 10 +
drivers/infiniband/hw/cxgb3/iwch_cm.c | 229 ++---
drivers/infiniband/hw/cxgb3/iwch_cm.h | 11 +-
4 files changed, 318 insertions(+), 48 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb3/iwch.c
b/drivers/infiniband/hw/cxgb3/iwch.c
index 0315c9d..da57b77 100644
--- a/drivers/infiniband/hw/cxgb3/iwch.c
+++ b/drivers/infiniband/hw/cxgb3/iwch.c
@@ -63,6 +63,115 @@ struct cxgb3_client t3c_client = {
static LIST_HEAD(dev_list);
static DEFINE_MUTEX(dev_mutex);
+static void insert_ifa(struct iwch_dev *rnicp, struct in_ifaddr *ifa)
+{
+ struct iwch_addrlist *addr;
+
+ addr = kmalloc(sizeof *addr, GFP_KERNEL);
+ if (!addr) {
+ printk(KERN_ERR MOD "%s - failed to alloc memory!\n",
+ __FUNCTION__);
+ return;
+ }
+ addr->ifa = ifa;
+ list_add_tail(&addr->entry, &rnicp->addrlist);
+}
+
+static void remove_ifa(struct iwch_dev *rnicp, struct in_ifaddr *ifa)
+{
+ struct iwch_addrlist *addr, *tmp;
+
+ list_for_each_entry_safe(addr, tmp, &rnicp->addrlist, entry) {
+ if (addr->ifa == ifa) {
+ list_del_init(&addr->entry);
+ kfree(addr);
+ return;
+ }
+ }
+}
+
+static int netdev_is_ours(struct iwch_dev *rnicp, struct net_device *netdev)
+{
+ int i;
+
+ for (i = 0; i < rnicp->rdev.port_info.nports; i++)
+ if (netdev == rnicp->rdev.port_info.lldevs[i])
+ return 1;
+ return 0;
+}
+
+static inline int is_iwarp_label(char *label)
+{
+ char *colon;
+
+ colon = strchr(label, ':');
+ if (colon && !strncmp(colon+1, "iw", 2))
+ return 1;
+ return 0;
+}
+
+static int nb_callback(struct notifier_block *self, unsigned long event,
+ void *ctx)
+{
+ struct in_ifaddr *ifa = ctx;
+ struct iwch_dev *rnicp = container_of(self, struct iwch_dev, nb);
+
+ printk(KERN_INFO "%s rnicp %p event %lx\n", __FUNCTION__, rnicp, event);
+
+ switch (event) {
+ case NETDEV_UP:
+ if (netdev_is_ours(rnicp, ifa->ifa_dev->dev) &&
+ is_iwarp_label(ifa->
