Re: Multiple USB NICs
On Tue, Nov 17, 2020 at 8:15 AM Mihai Popescu wrote: > Lee Nelson wrote: > > > If I have multiple USB Ethernet adapters of identical make and model, > > how does OpenBSD distinguish them over time. > > Is this happening to APU with same hardware interfaces, too? > No, 'wired in' interfaces are initialized in a predictable sequence. NICs installed in, e.g., PCI or PCIe slots also get consistent numbering. If you have multiple NICs installed the numbering might change if you remove one (i.e. fxp1 might become fxp0 if you remove the old fxp0). -ken
Re: Multiple USB NICs
Lee Nelson wrote: > If I have multiple USB Ethernet adapters of identical make and model, > how does OpenBSD distinguish them over time. Is this happening to APU with same hardware interfaces, too?
Re: Multiple USB NICs
On 2020-10-22, Lee Nelson wrote: > And Theo's hint was spot on. I'm experimenting with arm64 on an RPI 4. > Stability is not one of my expectations. I would actually expect pi4 and the onboard bse(4) to be fairly stable. Stability with USB is likely to be less good. (The onboard nic is also decently fast, I saw full gigabit near as dammit, much better than e.g. an APU, and also much better than the ure(4) which I tried on the pi, IIRC I was seeing a couple of hundred Mb tops from that).
Re: Multiple USB NICs
2020-10-22 07:35, Stuart Longland wrote: PCIe devices _can_ be connected to a Raspberry Pi 4, but it's a rather hap-hazard process that's not recommended unless you _really_ like re-working high-speed data links on printed circuit boards. Closest you get on a 'Pi is maybe some of the SPI Ethernet ICs that you might be able to hook to the GPIO header, but then you'd have to hack the OpenBSD kernel to support it, and it won't support gigabit speeds. That's not entirely true anymore. Just the other day the RPi Compute Module 4 was released. It exposes the PCI interface. From the Datasheet [1] 2.3. PCIe (Gen2 x1) The CM4 has an internal PCIe 2.0 x1 host controller. While on the Raspberry Pi 4, Model B this has been connected to a USB 3 host controller (using the Via Labs VLI805), on the CM4 the product designer is free to choose how the interface is used. A standard x86 machine and a multi-port network card is looking pretty good at this point. Yes, this still holds true. If low power consumption is not crucial, I too would rather go for a decent amd64 plattform. [1]: https://datasheets.raspberrypi.org/cm4/cm4-datasheet.pdf
Re: Multiple USB NICs
On 22/10/20 1:08 pm, Lee Nelson wrote: > The same sort of thing happened to me with me PCI cards, but it was > another edge case. I had two identical 2-port NIC's representing > em0-em3. The card with em0 and em1 died and brought the syste down with > a kernel panic. Upon rebooting the card that had been em2 and em3 was > now em0 and em1. The server could have still functioned on half the > ports but now the configuration was wrong for the surviving ports so the > server was unreachable. Yeah, the thing that's in PCI's favour is that it all gets power at the same time, whereas in USB, the bus gets powered up one hub at a time as each downstream hub is detected in the tree and powered up. Also the PCI bus is synchronised to a common clock, whereas USB is entirely asynchronous. Thus it's a lot easier to enforce some sort of order in PCI than USB. > And Theo's hint was spot on. I'm experimenting with arm64 on an RPI 4. > Stability is not one of my expectations. This is the normally standby > half of the fw pair of my home network. Even if it bursts into flames, > it will still be a learning experience. Yes well, it was in the back of my mind that this might be some sort of interface-challenged device. PCIe devices _can_ be connected to a Raspberry Pi 4, but it's a rather hap-hazard process that's not recommended unless you _really_ like re-working high-speed data links on printed circuit boards. Closest you get on a 'Pi is maybe some of the SPI Ethernet ICs that you might be able to hook to the GPIO header, but then you'd have to hack the OpenBSD kernel to support it, and it won't support gigabit speeds. A standard x86 machine and a multi-port network card is looking pretty good at this point. -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere.
Re: Multiple USB NICs
The same sort of thing happened to me with me PCI cards, but it was another edge case. I had two identical 2-port NIC's representing em0-em3. The card with em0 and em1 died and brought the syste down with a kernel panic. Upon rebooting the card that had been em2 and em3 was now em0 and em1. The server could have still functioned on half the ports but now the configuration was wrong for the surviving ports so the server was unreachable. This would likely happen on most OS's so the real moral of the story is don't use multiple, identical NIC's in your systems. There are probably other good reasons why this is a bad idea anyways. And Theo's hint was spot on. I'm experimenting with arm64 on an RPI 4. Stability is not one of my expectations. This is the normally standby half of the fw pair of my home network. Even if it bursts into flames, it will still be a learning experience. On Tue, 20 Oct 2020, Theo de Raadt wrote: Stuart Longland wrote: On 21/10/20 9:55 am, Lee Nelson wrote: Alternatively use a single nic with vlans, and break out to separate ports on a managed switch. Yes, that could work too, but this is one side of a pfsync/carp redundant firewall setup, so I want to keep it as simple as possible. Silly question, what hardware are the USB NICs plugging into? USB trades off determinism for hot-pluggability, and it seems a firewall, you absolutely do want an interface to appear in a specific location. I'd be looking at something that plugs into the system peripheral bus somehow (PCIe, PCI, ISA, … etc). Oh come on, you know the answer before you ask it. Using cheap hardware and expecting free software developers to pull magic out of their ass to make it solve unsolveable problems, and produce a result as too as state of the art expensive hardware --- or even cheaper hardware --- with DEDICATED PORTS -- it is madness. We can't do it. And we said so. And Lee gets it. But do the rest of the thread participants? I think it's fine for us as a community to humour the attempt for a bit, but THEN THE DISCUSSION MIGHT AS WELL END, as the consequences of the choice ARE WHAT THEY ARE. You get what you paid for. And we (OpenBSD) played no part in the decision or the consequences, hotplug is what it is. Can we end this discussion?
Re: Multiple USB NICs
I thought we were free to worship our totalitarian leader, butt an all, on and off list. This is, after all, not a linux list. :) Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Wednesday, 21 October 2020 22:08, Stuart Longland wrote: > On 21/10/20 10:53 pm, pipus wrote: > > > but Theo your butt is magical :( > > Perhaps you can worship it off list then. ;-) > > > > Stuart Longland (aka Redhatter, VK4MSL) > > I haven't lost my mind... > ...it's backed up on a tape somewhere.
Re: Multiple USB NICs
On 21/10/20 10:53 pm, pipus wrote: > but Theo your butt is magical :( Perhaps you can worship it off list then. ;-) -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere.
Re: Multiple USB NICs
but Theo your butt is magical :( You do it no justice. I have a microwave that is a bit glitchy . Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Wednesday, 21 October 2020 07:42, Theo de Raadt wrote: > Stuart Longland stua...@longlandclan.id.au wrote: > > > On 21/10/20 9:55 am, Lee Nelson wrote: > > > > > > Alternatively use a single nic with vlans, and break out to separate > > > > ports on a managed switch. > > > > > > Yes, that could work too, but this is one side of a pfsync/carp > > > redundant firewall setup, so I want to keep it as simple as possible. > > > > Silly question, what hardware are the USB NICs plugging into? > > USB trades off determinism for hot-pluggability, and it seems a > > firewall, you absolutely do want an interface to appear in a specific > > location. I'd be looking at something that plugs into the system > > peripheral bus somehow (PCIe, PCI, ISA, … etc). > > Oh come on, you know the answer before you ask it. > > Using cheap hardware and expecting free software developers to > pull magic out of their ass to make it solve unsolveable problems, and > produce a result as too as state of the art expensive hardware --- or > even cheaper hardware --- with DEDICATED PORTS -- it is madness. We > can't do it. And we said so. > > And Lee gets it. But do the rest of the thread participants? > > I think it's fine for us as a community to humour the attempt for a bit, > but THEN THE DISCUSSION MIGHT AS WELL END, as the consequences of the > choice ARE WHAT THEY ARE. > > You get what you paid for. And we (OpenBSD) played no part in the > decision or the consequences, hotplug is what it is. > > Can we end this discussion?
Re: Multiple USB NICs
Stuart Longland wrote: > On 21/10/20 9:55 am, Lee Nelson wrote: > >> Alternatively use a single nic with vlans, and break out to separate > >> ports on a managed switch. > >> > > Yes, that could work too, but this is one side of a pfsync/carp > > redundant firewall setup, so I want to keep it as simple as possible. > > Silly question, what hardware are the USB NICs plugging into? > > USB trades off determinism for hot-pluggability, and it seems a > firewall, you absolutely do want an interface to appear in a specific > location. I'd be looking at something that plugs into the system > peripheral bus somehow (PCIe, PCI, ISA, … etc). Oh come on, you know the answer before you ask it. Using cheap hardware and expecting free software developers to pull magic out of their ass to make it solve unsolveable problems, and produce a result as too as state of the art expensive hardware --- or even cheaper hardware --- with DEDICATED PORTS -- it is madness. We can't do it. And we said so. And Lee gets it. But do the rest of the thread participants? I think it's fine for us as a community to humour the attempt for a bit, but THEN THE DISCUSSION MIGHT AS WELL END, as the consequences of the choice ARE WHAT THEY ARE. You get what you paid for. And we (OpenBSD) played no part in the decision or the consequences, hotplug is what it is. Can we end this discussion?
Re: Multiple USB NICs
On 21/10/20 9:55 am, Lee Nelson wrote: >> Alternatively use a single nic with vlans, and break out to separate >> ports on a managed switch. >> > Yes, that could work too, but this is one side of a pfsync/carp > redundant firewall setup, so I want to keep it as simple as possible. Silly question, what hardware are the USB NICs plugging into? USB trades off determinism for hot-pluggability, and it seems a firewall, you absolutely do want an interface to appear in a specific location. I'd be looking at something that plugs into the system peripheral bus somehow (PCIe, PCI, ISA, … etc). -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere.
Re: Multiple USB NICs
On Tue, 20 Oct 2020, Stuart Henderson wrote: On 2020-10-20, Lee Nelson wrote: The only real solution here, aside from using better hardware, seems to be to use adapters with different drivers. That is the approach I'm trying next. Alternatively use a single nic with vlans, and break out to separate ports on a managed switch. Yes, that could work too, but this is one side of a pfsync/carp redundant firewall setup, so I want to keep it as simple as possible.
Re: Multiple USB NICs
On 2020-10-20, Lee Nelson wrote: > The only real solution here, aside from using better hardware, seems to be > to use adapters with different drivers. That is the approach I'm trying > next. Alternatively use a single nic with vlans, and break out to separate ports on a managed switch.
Re: Multiple USB NICs
On Tue, 20 Oct 2020, Aaron Mason wrote: On Tue, Oct 20, 2020 at 12:29 PM Lee Nelson wrote: On Mon, 19 Oct 2020, Allan Streib wrote: Lee Nelson writes: I had considered some late-running script that would query the MAC's of each NIC and then configure them accordingly or rewrite the hostname.* files and call netstart on them, but that just seems sloppy and unreliable. What about DHCP? It supports MAC-specific configurations. Allan Very good point, except in my case, this machine would be the dhcp server. You could run scripts to set the IP address based on the MAC with this in each hostname.axenN file: !/path/to/script axeN And then write that script to pick the IP address based on the MAC. -- Aaron Mason - Programmer, open source addict I've taken my software vows - for beta or for worse Brilliant! That's the kind of thing I normally think of, but I hadn't yet. I wish I had thought of this before I ordered adapters using different drivers on impulse. Still, this is an idea I will probably use at some point. Thank you!
Re: Multiple USB NICs
On Tue, Oct 20, 2020 at 12:29 PM Lee Nelson wrote: > > > > On Mon, 19 Oct 2020, Allan Streib wrote: > > > Lee Nelson writes: > > > >> I had considered some late-running script that would query the MAC's of > >> each NIC and then configure them accordingly or rewrite the hostname.* > >> files and call netstart on them, but that just seems sloppy and > >> unreliable. > > > > What about DHCP? It supports MAC-specific configurations. > > > > Allan > > > Very good point, except in my case, this machine would be the dhcp server. > You could run scripts to set the IP address based on the MAC with this in each hostname.axenN file: !/path/to/script axeN And then write that script to pick the IP address based on the MAC. -- Aaron Mason - Programmer, open source addict I've taken my software vows - for beta or for worse
Re: Multiple USB NICs
On Mon, 19 Oct 2020, Allan Streib wrote: Lee Nelson writes: I had considered some late-running script that would query the MAC's of each NIC and then configure them accordingly or rewrite the hostname.* files and call netstart on them, but that just seems sloppy and unreliable. What about DHCP? It supports MAC-specific configurations. Allan Very good point, except in my case, this machine would be the dhcp server.
Re: Multiple USB NICs
Lee Nelson writes: > I had considered some late-running script that would query the MAC's of > each NIC and then configure them accordingly or rewrite the hostname.* > files and call netstart on them, but that just seems sloppy and > unreliable. What about DHCP? It supports MAC-specific configurations. Allan
Re: Multiple USB NICs
On Mon, 19 Oct 2020, Theo de Raadt wrote: Lee Nelson wrote: If I have multiple USB Ethernet adapters of identical make and model, how does OpenBSD distinguish them over time. In the order their drivers reach "interface attach" code. There are multiple reasons the drivers could reach this out of order. In other words if there's a reboot or the adapters get unplugged and moved around and plugged back in, is there a way to make sure that the adapters associated with axen0 and axen1 maintain those associations? No. Is there some way to pin an interface to the adapters MAC address? No method comes to mind. Well there is a method, but it is incompatible with KARL kernel relinking so I don't think people should use that, and hence I'm not going to explain it. A related question: if the adapters never move, and always stay in the same place on their respective USB hubs, will they always be numbered the same (axen0, axen1, etc)? Cannot gaurantee that. Especially if they are on different usb busses, or the initial probe fails, some probes may occur late and thus adjust the order. From my testing, all the adapters are plugged into the same hub (not a good idea) and they don't get moved around, they appear to stay in order. I suppose that could depend on the quality of the hub being used. However, in the case of my RPI4, I have a USB3 hub connected to the root USB xHCI hub. If I move an adapter from that hub to one of the USB 2.0 ports (which are themselves a hub), then it's there's no real pattern was to which will come first, the adapters on the USB 3 hub or the adapter on the internal USB 2 hub. The only real solution here, aside from using better hardware, seems to be to use adapters with different drivers. That is the approach I'm trying next. I had considered some late-running script that would query the MAC's of each NIC and then configure them accordingly or rewrite the hostname.* files and call netstart on them, but that just seems sloppy and unreliable.
Re: Multiple USB NICs
Lee Nelson wrote: > If I have multiple USB Ethernet adapters of identical make and model, > how does OpenBSD distinguish them over time. In the order their drivers reach "interface attach" code. There are multiple reasons the drivers could reach this out of order. > In other words if > there's a reboot or the adapters get unplugged and moved around and > plugged back in, is there a way to make sure that the adapters > associated with axen0 and axen1 maintain those associations? No. > Is there some way to pin an interface to the adapters MAC address? No method comes to mind. Well there is a method, but it is incompatible with KARL kernel relinking so I don't think people should use that, and hence I'm not going to explain it. > A related question: if the adapters never move, and always stay in the > same place on their respective USB hubs, will they always be numbered > the same (axen0, axen1, etc)? Cannot gaurantee that. Especially if they are on different usb busses, or the initial probe fails, some probes may occur late and thus adjust the order.
Multiple USB NICs
If I have multiple USB Ethernet adapters of identical make and model, how does OpenBSD distinguish them over time. In other words if there's a reboot or the adapters get unplugged and moved around and plugged back in, is there a way to make sure that the adapters associated with axen0 and axen1 maintain those associations? Is there some way to pin an interface to the adapters MAC address? A related question: if the adapters never move, and always stay in the same place on their respective USB hubs, will they always be numbered the same (axen0, axen1, etc)?
