In case someone finds this thread in the future, I would like to add that I have now received a possible solution to the problem out-of-band. The solution is to use pair(4):
The following setup works for me, although it is a bit too convoluted: # cat /etc/hostname.pair0 up # cat /etc/hostname.pair1 rdomain 1 up patch pair0 # cat /etc/hostname.bridge0 add em2 add vether0 add pair0 !dhclient vether0 !dhclient pair1 In this setup, both vether0 and pair1 gets a separate IP address on the same network but in different routing domains, but they can still talk to each other over the patched pair+bridge. This leads to the following mess, only possible to decipher with a monospace font: .----------------.---------. 10.0.0.1 .-,( ),-. | bridge0 | em2 | dhcp server .-( )-. | | (no ip) |---> gateway --->( internet ) | '---------' __________ '-( ).-' .-----------. .-----------.| [_...__...°] '-.( ).-' | vether0 | | pair0 || | rdomain 0 | | rdomain 0 || | dhcp | | (no ip) || | 10.0.0.2 | '-----------'| .-----------. '-----------'-------^------' | pair1 | | | rdomain 1 | '---patch---| dhcp | | 10.0.0.3 | '-----------' Still not sure if this is a good idea, but it is the solution to my literal problem so I consider that one solved. On Sat, Apr 22, 2017 at 3:49 AM, Anders Andersson <pipat...@gmail.com> wrote: > === BACKGROUND === > > I'm trying to set up an OpenBSD 6.1 server having two externally visible > IP numbers through one physical network port, each IP mapping to a > unique MAC address[1]. I have it mostly working, but my interfaces can't > talk to each other. > > All traffic should use the primary IP, and most services should listen > on that. The secondary IP should only be used on-demand for one or two > services. > > Thinking that separate routing tables can solve this, I have configured > my network like this[2][3]: > > # cat hostname.em2 > up > > # cat hostname.vether0 > lladdr 00:00:00:00:00:02 > > # cat hostname.vether1 > lladdr 00:00:00:00:00:03 rdomain 1 > > # cat hostname.bridge0 > add em2 add vether0 add vether1 up > !dhclient vether0 > !dhclient vether1 > > # cat sysctl.conf > net.inet.ip.forwarding=1 > > Leading to something like this[4]: > (full post in monospace: http://paste.debian.net/928811 ) > > .----------------.---------. 10.0.0.1 .-,( ),-. > | bridge0 | em2 | dhcp server .-( )-. > | | (no ip) |---> gateway --->( internet ) > | '---------' __________ '-( ).-' > | | [_...__...°] '-.( ).-' > .-----------. .-----------. > | vether0 | | vether1 | > | rdomain 0 | | rdomain 1 | > | dhcp | | dhcp | > | 10.0.0.2 | | 10.0.0.3 | > '-----------'--'-----------' > > Everything else should be the default, this is on a clean 6.1 install. > > This configuration works great, vether0 and vether1 both gets an IP > number from my DHCP server, all traffic goes out on vether0 by default, > but I can select vether1 manually: > > # traceroute -nvq1 10.0.0.1 > traceroute to 10.0.0.1 (10.0.0.1), 64 hops max, 40 byte packets > 1 10.0.0.1 48 bytes to 10.0.0.2 0.994 ms > > # route -T1 exec traceroute -nvq1 10.0.0.1 > traceroute to 10.0.0.1 (10.0.0.1), 64 hops max, 40 byte packets > 1 10.0.0.1 48 bytes to 10.0.0.3 0.984 ms > > I can also reach each IP from outside the box. They are going in on em2, > through the bridge, and in to vether0 or vether1 respectively. > > > > > === PROBLEM === > > Now to my problem: I have no connection between vether0<->vether1. > > # traceroute -nvq1 10.0.0.3 > traceroute to 10.0.0.3 (10.0.0.3), 64 hops max, 40 byte... > 1 * > 2 * > ^C > > If I listen with tcpdump on the bridge, I see lots of unanswered arp > who-has: > > # tcpdump -nti bridge0 > tcpdump: listening on bridge0, link-type EN10MB > arp who-has 10.0.0.3 tell 10.0.0.2 > arp who-has 10.0.0.3 tell 10.0.0.2 > ^C > > These packets even go out on em2 to my LAN, but no one ever answers. The > same thing happens in reverse. > > I have experimented with these bridge settings: > 'blocknonip' - adding or removing on members makes no difference > 'discover' - should be the default, adding makes no difference > 'learn' - should be the default, adding makes no difference > > > > === EXPECTATIONS === > > I expected that someone should answer those arp who-is requests, either > vether1 directly, or the bridge0 who should know which interfaces it > has. Is there something I must configure to make this work, or is my > plan flawed from the start? > > > > > > === INFORMATION === > > Various information that could help answer my question (trimmed > whitespace and boilerplate): > > # route -n show -inet > Destination Gateway Flags Refs Use Mtu Prio Iface > default 10.0.0.1 UGS 0 0 - 8 vether0 > 224/4 127.0.0.1 URS 0 0 32768 8 lo0 > 10.0.0/24 10.0.0.2 UCn 1 0 - 4 vether0 > 10.0.0.1 link#6 UHLch 1 1 - 3 vether0 > 10.0.0.2 00:00:00:00:00:02 UHLl 0 0 - 1 vether0 > 10.0.0.255 10.0.0.2 UHb 0 0 - 1 vether0 > 127/8 127.0.0.1 UGRS 0 0 32768 8 lo0 > 127.0.0.1 127.0.0.1 UHhl 1 2 32768 1 lo0 > > # route -T1 -n show -inet > Destination Gateway Flags Refs Use Mtu Prio Iface > default 10.0.0.1 UGS 0 32 - 8 vether1 > 10.0.0/24 10.0.0.3 UCn 1 4 - 4 vether1 > 10.0.0.1 00:00:00:00:00:01 UHLch 1 3 - 3 vether1 > 10.0.0.3 00:00:00:00:00:03 UHLl 0 0 - 1 vether1 > 10.0.0.255 10.0.0.3 UHb 0 0 - 1 vether1 > > # for if in bridge0 em2 vether{0,1}; do ifconfig $if; done > bridge0: flags=41<UP,RUNNING> > description: Bridge for external virtual NICs > index 9 llprio 3 > groups: bridge > priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp > designated: id 00:00:00:00:00:00 priority 0 > em2 flags=3<LEARNING,DISCOVER> > port 3 ifpriority 0 ifcost 0 > vether0 flags=3<LEARNING,DISCOVER> > port 6 ifpriority 0 ifcost 0 > vether1 flags=3<LEARNING,DISCOVER> > port 7 ifpriority 0 ifcost 0 > Addresses (max cache: 100, timeout: 240): > 00:00:00:00:00:01 em2 1 flags=0<> > em2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > mtu 1500 > lladdr xx:xx:xx:xx:xx:xx > description: External interface #1 (integrated NIC) > index 3 priority 0 llprio 3 > media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause) > status: active > vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 > lladdr 00:00:00:00:00:02 > description: Main external IP > index 6 priority 0 llprio 3 > groups: vether egress > media: Ethernet autoselect > status: active > inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 > vether1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> > rdomain 1 mtu 1500 > lladdr 00:00:00:00:00:03 > description: Secondary external IP > index 7 priority 0 llprio 3 > groups: vether > media: Ethernet autoselect > status: active > inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255 > > > > > > Footnotes: > > [1] My ISP implements carrier-grade NAT, with an opt-out. If I supply a > MAC address, they will whitelist it and give it an external IP > number over DHCP. I can do this for more than one card but they must > have a unique MAC, meaning that I can not simply use an IP alias. > While experimenting with the configuration, I am using my own > internal DHCP server. The plan is to connect the OpenBSD server > directly to the ISP. > > [2] It should not be necessary to have two vether, em2 should be able to > serve the purpose of the primary interface. However, I think I'm > hitting the bug described in the first paragraph here: > http://undeadly.org/cgi?action=article&sid=20160725144108 > > [3] The MAC addresses in this report are obfuscated: 10.0.0.xx maps to > 00:00:00:00:00:xx to aid connecting a card with its IP. The virtual > interfaces has valid MAC addresses taken from retired NICs. > > [4] http://search.cpan.org/dist/App-Asciio/lib/App/Asciio.pm