Re: Manually "naming" Multiple NICs
I think the following tip is worth repeating- "I still recommend writing the last few digits of the MAC addr on the spine of the NIC" Such a simple thing to do, and it will save a lot of time and confusion... Dan Farrell Applied Innovations [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Nick Holland Sent: Thursday, May 11, 2006 7:44 PM To: misc Subject: Re: Manually "naming" Multiple NICs [EMAIL PROTECTED]@mgEDV.net wrote: [you edited out discussion of *USB* devices] >> Normally these devices come up in the same order each time. >> >> It is not gauranteed, unfortunately, because device bring up can >> race against other devices. I've seen it be non-deterministic. >> > > me, too. especially, if you plug in another nic on pci between 2 > other nics. this is really confusing the box. also take care for > your bios interrupt settings - if you have a lot of traffic, it > sometimes can be smart to put all the nics on the same interrupt. You creatively edited out the part that clearly indicated the topic of the part you quoted was USB devices, not PCI. PCI devices come up very predictably, if you understand how it works. Nothing is confusing to the box at all. YOU may be confused, but the box is quite sure of what it is doing. :) A little experimentation (which everyone should do before putting a box into production) will make it very clear. Assuming you have all the same kind of NIC, it goes by slot number. Yes, if you stuff a card between two other cards, yes, the cards are EXPECTED to change IDs, I can't think of anything else I would want it to do (being that I really hate it when software or hardware tries to make guesses about what I was intending). If that isn't what you want, first move one of the existing cards, then add the new card to the vacated slot. I still recommend writing the last few digits of the MAC addr on the spine of the NIC, as some machines number their slots in a curious way. A Dell GX1 will number its slots something like "2 3 4 0 1". BUT...if you blow a NIC, pull one out and put a spare in, all stays as it was. If you change it out for a different type of NIC, things will change, but you can easily predict what they will end up being, and if you don't understand what is going on, dmesg and the label I suggested putting on the card will clear it all up. :) If you pull the NICs out of one machine and drop in another, you have to take a little care to make sure you know where they end up, but again, once you know where they are, they stay put, and you can replace them and know what will end up happening predictively. Michael Schmidt wrote: > Please allow a silly question: What4s the reason for "put all the > nics on the same interrupt" if one has a lot of traffic? First, you start messing with your BIOS at that level, you are more likely to break things than improve things. The way people usually shoot themselves in the foot is by forgetting that PCI was designed to share interrupts, and trying to use the old ISA logic of "each device on its own IRQ". Here is a massively over-simplified explaination of what happens: Let's say you have five PCI devices which could trigger interrupts. Again, the PCI bus was designed to share IRQs, so when an IRQ comes through, the first thing that happens is you have to push a bunch of stuff off to the stack before servicing the IRQ, then you have to identify which device fired off the IRQ before you can do anything about it. If a lot of IRQs are happening, so the next is often coming in before the previous one was finished, there are two ways things could happen: 1) You return to what you were doing before the IRQ came in, but then the IRQ fires again, sending you back to the IRQ service routine. (this is how unshared ISA IRQs were handled) 2) BEFORE returning, you check to see if any other device needs to be serviced, which is possible if they are sharing the same IRQ channel. In fact, if you are sharing the same IRQ, you have to do this. Hopefully, it is moderately clear that you save two IRQ-caused context switches by staying in the IRQ service routine and looking for more work to do. The more additional IRQs that come in, the more potential savings. This is possible if you are sharing IRQs. IRQs are costly things when it comes to processor time. Reality check #1: This only matters on HIGHLY loaded systems, where IRQs are stacking up. If your system is this loaded and you have less than the best HW, you can probably get better gains by selecting different hardware. A machine which has multiple PCI buses or higher-quality NICs will probably give you a MUCH larger benefit than IRQ sharing. My wild-a**ed guess of what IRQ sharing would do for you would be at most a few percentage points, not enough that anyone would eve
Re: Manually "naming" Multiple NICs
[EMAIL PROTECTED]@mgEDV.net wrote: [you edited out discussion of *USB* devices] Normally these devices come up in the same order each time. It is not gauranteed, unfortunately, because device bring up can race against other devices. I've seen it be non-deterministic. me, too. especially, if you plug in another nic on pci between 2 other nics. this is really confusing the box. also take care for your bios interrupt settings - if you have a lot of traffic, it sometimes can be smart to put all the nics on the same interrupt. You creatively edited out the part that clearly indicated the topic of the part you quoted was USB devices, not PCI. PCI devices come up very predictably, if you understand how it works. Nothing is confusing to the box at all. YOU may be confused, but the box is quite sure of what it is doing. :) A little experimentation (which everyone should do before putting a box into production) will make it very clear. Assuming you have all the same kind of NIC, it goes by slot number. Yes, if you stuff a card between two other cards, yes, the cards are EXPECTED to change IDs, I can't think of anything else I would want it to do (being that I really hate it when software or hardware tries to make guesses about what I was intending). If that isn't what you want, first move one of the existing cards, then add the new card to the vacated slot. I still recommend writing the last few digits of the MAC addr on the spine of the NIC, as some machines number their slots in a curious way. A Dell GX1 will number its slots something like "2 3 4 0 1". BUT...if you blow a NIC, pull one out and put a spare in, all stays as it was. If you change it out for a different type of NIC, things will change, but you can easily predict what they will end up being, and if you don't understand what is going on, dmesg and the label I suggested putting on the card will clear it all up. :) If you pull the NICs out of one machine and drop in another, you have to take a little care to make sure you know where they end up, but again, once you know where they are, they stay put, and you can replace them and know what will end up happening predictively. Michael Schmidt wrote: Please allow a silly question: What4s the reason for "put all the > nics on the same interrupt" if one has a lot of traffic? First, you start messing with your BIOS at that level, you are more likely to break things than improve things. The way people usually shoot themselves in the foot is by forgetting that PCI was designed to share interrupts, and trying to use the old ISA logic of "each device on its own IRQ". Here is a massively over-simplified explaination of what happens: Let's say you have five PCI devices which could trigger interrupts. Again, the PCI bus was designed to share IRQs, so when an IRQ comes through, the first thing that happens is you have to push a bunch of stuff off to the stack before servicing the IRQ, then you have to identify which device fired off the IRQ before you can do anything about it. If a lot of IRQs are happening, so the next is often coming in before the previous one was finished, there are two ways things could happen: 1) You return to what you were doing before the IRQ came in, but then the IRQ fires again, sending you back to the IRQ service routine. (this is how unshared ISA IRQs were handled) 2) BEFORE returning, you check to see if any other device needs to be serviced, which is possible if they are sharing the same IRQ channel. In fact, if you are sharing the same IRQ, you have to do this. Hopefully, it is moderately clear that you save two IRQ-caused context switches by staying in the IRQ service routine and looking for more work to do. The more additional IRQs that come in, the more potential savings. This is possible if you are sharing IRQs. IRQs are costly things when it comes to processor time. Reality check #1: This only matters on HIGHLY loaded systems, where IRQs are stacking up. If your system is this loaded and you have less than the best HW, you can probably get better gains by selecting different hardware. A machine which has multiple PCI buses or higher-quality NICs will probably give you a MUCH larger benefit than IRQ sharing. My wild-a**ed guess of what IRQ sharing would do for you would be at most a few percentage points, not enough that anyone would ever notice in the real world of "how it feels to the user" results. Reality check #2: I'm not a kernel hacker. But in eight+ hours, no one else answered. :) If you take what I said too seriously, you are nuttier than I am for saying it. Nick.
Re: Manually "naming" Multiple NICs
[EMAIL PROTECTED]@mgEDV.net wrote: me, too. especially, if you plug in another nic on pci between 2 other nics. this is really confusing the box. also take care for your bios interrupt settings - if you have a lot of traffic, it sometimes can be smart to put all the nics on the same interrupt. Please allow a silly question: What4s the reason for "put all the nics on the same interrupt" if one has a lot of traffic? Have a nice day Michael -- Michael Schmidt MIRRORS: DJGPP ftp://ftp.fh-koblenz.de/pub/DJGPP/ Ghostscript ftp://ftp.fh-koblenz.de/pub/Ghostscript/
Re: Manually "naming" Multiple NICs
> Normally these devices come up in the same order each time. > > It is not gauranteed, unfortunately, because device bring up can > race against other devices. I've seen it be non-deterministic. > me, too. especially, if you plug in another nic on pci between 2 other nics. this is really confusing the box. also take care for your bios interrupt settings - if you have a lot of traffic, it sometimes can be smart to put all the nics on the same interrupt.
Re: Manually "naming" Multiple NICs
On 5/9/06, Dave Crawford <[EMAIL PROTECTED]> wrote: Greetings, I've been running a Debian based firewall for a number of years and have a need to update the hardware, so have decided to change the OS over to OpenBSD at the same time. The box has 4 NICs(identical make/model) and by using 'ifrename' I'm able to specify which NICs are assigned each particular iface name (i.e. eth1, eth2) by MAC address. This alleviates the concern of the kernel assigning 'eth0' (for example) to the improper NIC when rebooting. I *hate* the linux eth0 thing. The OpenBSD method is starts with a slot on the pci bus and names your nics in order. The OpenBSD method is much cleaner. I think you'll like it once it makes sense to you. --Bryan
Re: Manually "naming" Multiple NICs
> > There may be a race in usb for how devices respond, but I bet it is small > > and not really that worrying. One day maybe someone can look at it. > > Is there something to look for there? My limited experience with usb on > OpenBSD leads me to think every thing comes in with the same order by > where things are plugged in, or where they occur internally (I've got 4 > sockets and an internal ugen(4)). Normally these devices come up in the same order each time. It is not gauranteed, unfortunately, because device bring up can race against other devices. I've seen it be non-deterministic.
Re: Manually "naming" Multiple NICs
On Tue, May 09, 2006 at 06:46:54PM -0600, Theo de Raadt wrote: > >So the order in which the cards are detected is deterministic and never > >changes? I'm not being a smartass, I really want to know. > > We have worked very hard at this. On a particular machine, yes, it should > be deterministic. On each bus, we probe in bus-order. For instance, pci > or isa, or pcmcia, or cardbus. Slots, connectors, functions, etc. Very cool. One more thing to like. > There may be a race in usb for how devices respond, but I bet it is small > and not really that worrying. One day maybe someone can look at it. Is there something to look for there? My limited experience with usb on OpenBSD leads me to think every thing comes in with the same order by where things are plugged in, or where they occur internally (I've got 4 sockets and an internal ugen(4)). -- Darrin Chandler| Phoenix BSD Users Group [EMAIL PROTECTED] | http://bsd.phoenix.az.us/ http://www.stilyagin.com/ |
Re: Manually "naming" Multiple NICs
On Tue, May 09, 2006 at 08:03:41PM -0500, Tony Abernethy wrote: > Darrin Chandler wrote: > > So the order in which the cards are detected is deterministic and never > > changes? I'm not being a smartass, I really want to know. > Me, I just lurk here, but the impression I get is that > 1) anywhere else you'd be a smartass. *Everywhere* else I'm a smartass. Won't happen here, though. -- Darrin Chandler| Phoenix BSD Users Group [EMAIL PROTECTED] | http://bsd.phoenix.az.us/ http://www.stilyagin.com/ |
Re: Manually "naming" Multiple NICs
Darrin Chandler wrote: > > On Tue, May 09, 2006 at 08:14:06PM -0400, Adam wrote: > > On Tue, 09 May 2006 19:52:10 -0400 Dave Crawford > <[EMAIL PROTECTED]> wrote: > > > > > or another viable solution. > > > > There's no solution because there's no problem. OpenBSD > doesn't randomly > > reorder interfaces for no reason. > > So the order in which the cards are detected is deterministic and never > changes? I'm not being a smartass, I really want to know. > > -- > Darrin Chandler| Phoenix BSD Users Group > [EMAIL PROTECTED] | http://bsd.phoenix.az.us/ > http://www.stilyagin.com/ | Me, I just lurk here, but the impression I get is that 1) anywhere else you'd be a smartass. 2) OpenBSD does NOT believe in screwing things up to make something simple a wee bit easier for the simple cases (forcing the complicated to forever be hopelessly confused) Watch the fun trying to refix the fixes ;) Methinks that making something simpler than it can be ... backfires.
Re: Manually "naming" Multiple NICs
>> There's no solution because there's no problem. OpenBSD doesn't randomly >> reorder interfaces for no reason. > >So the order in which the cards are detected is deterministic and never >changes? I'm not being a smartass, I really want to know. We have worked very hard at this. On a particular machine, yes, it should be deterministic. On each bus, we probe in bus-order. For instance, pci or isa, or pcmcia, or cardbus. Slots, connectors, functions, etc. There may be a race in usb for how devices respond, but I bet it is small and not really that worrying. One day maybe someone can look at it. One card is special, ep(4) because of plug & play. But I bet that is not what anyone is concerned about. But even that man page lays out the probe order...
Re: Manually "naming" Multiple NICs
>I've been running a Debian based firewall for a number of years and have >a need to update the hardware, so have decided to change the OS over to >OpenBSD at the same time. > >The box has 4 NICs(identical make/model) and by using 'ifrename' I'm >able to specify which NICs are assigned each particular iface name (i.e. >eth1, eth2) by MAC address. This alleviates the concern of the kernel >assigning 'eth0' (for example) to the improper NIC when rebooting. In OpenBSD, we have made sure that all our busses are deterministically probed. Therefore these devices will always come up with the same names, every time. >After 'googling' for awhile I've found that 'ifrename' is available in >FreeBSD but I was unable to find confirmation that it's available for >OpenBSD, or another viable solution. We don't have such a renaming. We do however have grouping for interfaces, if you want to refer to them via alternative names in pf and such subsystems. man ifconfig, and read up on the group and -group keywords.
Re: Manually "naming" Multiple NICs
On Tue, May 09, 2006 at 08:14:06PM -0400, Adam wrote: > On Tue, 09 May 2006 19:52:10 -0400 Dave Crawford <[EMAIL PROTECTED]> wrote: > > > or another viable solution. > > There's no solution because there's no problem. OpenBSD doesn't randomly > reorder interfaces for no reason. So the order in which the cards are detected is deterministic and never changes? I'm not being a smartass, I really want to know. -- Darrin Chandler| Phoenix BSD Users Group [EMAIL PROTECTED] | http://bsd.phoenix.az.us/ http://www.stilyagin.com/ |
Re: Manually "naming" Multiple NICs
On Tue, 09 May 2006 19:52:10 -0400 Dave Crawford <[EMAIL PROTECTED]> wrote: > or another viable solution. There's no solution because there's no problem. OpenBSD doesn't randomly reorder interfaces for no reason. Adam
Manually "naming" Multiple NICs
Greetings, I've been running a Debian based firewall for a number of years and have a need to update the hardware, so have decided to change the OS over to OpenBSD at the same time. The box has 4 NICs(identical make/model) and by using 'ifrename' I'm able to specify which NICs are assigned each particular iface name (i.e. eth1, eth2) by MAC address. This alleviates the concern of the kernel assigning 'eth0' (for example) to the improper NIC when rebooting. After 'googling' for awhile I've found that 'ifrename' is available in FreeBSD but I was unable to find confirmation that it's available for OpenBSD, or another viable solution. Any pointers are greatly appreciated. -Dave
