=== BACKGROUND ===
I'm trying to set up an OpenBSD 6.1 server having two externally visible
IP numbers through one physical network port, each IP mapping to a
unique MAC address[1]. I have it mostly working, but my interfaces can't
talk to each other.
All traffic should use the primary IP, and most services should listen
on that. The secondary IP should only be used on-demand for one or two
services.
Thinking that separate routing tables can solve this, I have configured
my network like this[2][3]:
# cat hostname.em2
up
# cat hostname.vether0
lladdr 00:00:00:00:00:02
# cat hostname.vether1
lladdr 00:00:00:00:00:03 rdomain 1
# cat hostname.bridge0
add em2 add vether0 add vether1 up
!dhclient vether0
!dhclient vether1
# cat sysctl.conf
net.inet.ip.forwarding=1
Leading to something like this[4]:
(full post in monospace: http://paste.debian.net/928811 )
.----------------.---------. 10.0.0.1 .-,( ),-.
| bridge0 | em2 | dhcp server .-( )-.
| | (no ip) |---> gateway --->( internet )
| '---------' __________ '-( ).-'
| | [_...__...°] '-.( ).-'
.-----------. .-----------.
| vether0 | | vether1 |
| rdomain 0 | | rdomain 1 |
| dhcp | | dhcp |
| 10.0.0.2 | | 10.0.0.3 |
'-----------'--'-----------'
Everything else should be the default, this is on a clean 6.1 install.
This configuration works great, vether0 and vether1 both gets an IP
number from my DHCP server, all traffic goes out on vether0 by default,
but I can select vether1 manually:
# traceroute -nvq1 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 64 hops max, 40 byte packets
1 10.0.0.1 48 bytes to 10.0.0.2 0.994 ms
# route -T1 exec traceroute -nvq1 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 64 hops max, 40 byte packets
1 10.0.0.1 48 bytes to 10.0.0.3 0.984 ms
I can also reach each IP from outside the box. They are going in on em2,
through the bridge, and in to vether0 or vether1 respectively.
=== PROBLEM ===
Now to my problem: I have no connection between vether0<->vether1.
# traceroute -nvq1 10.0.0.3
traceroute to 10.0.0.3 (10.0.0.3), 64 hops max, 40 byte...
1 *
2 *
^C
If I listen with tcpdump on the bridge, I see lots of unanswered arp
who-has:
# tcpdump -nti bridge0
tcpdump: listening on bridge0, link-type EN10MB
arp who-has 10.0.0.3 tell 10.0.0.2
arp who-has 10.0.0.3 tell 10.0.0.2
^C
These packets even go out on em2 to my LAN, but no one ever answers. The
same thing happens in reverse.
I have experimented with these bridge settings:
'blocknonip' - adding or removing on members makes no difference
'discover' - should be the default, adding makes no difference
'learn' - should be the default, adding makes no difference
=== EXPECTATIONS ===
I expected that someone should answer those arp who-is requests, either
vether1 directly, or the bridge0 who should know which interfaces it
has. Is there something I must configure to make this work, or is my
plan flawed from the start?
=== INFORMATION ===
Various information that could help answer my question (trimmed
whitespace and boilerplate):
# route -n show -inet
Destination Gateway Flags Refs Use Mtu Prio Iface
default 10.0.0.1 UGS 0 0 - 8 vether0
224/4 127.0.0.1 URS 0 0 32768 8 lo0
10.0.0/24 10.0.0.2 UCn 1 0 - 4 vether0
10.0.0.1 link#6 UHLch 1 1 - 3 vether0
10.0.0.2 00:00:00:00:00:02 UHLl 0 0 - 1 vether0
10.0.0.255 10.0.0.2 UHb 0 0 - 1 vether0
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHhl 1 2 32768 1 lo0
# route -T1 -n show -inet
Destination Gateway Flags Refs Use Mtu Prio Iface
default 10.0.0.1 UGS 0 32 - 8 vether1
10.0.0/24 10.0.0.3 UCn 1 4 - 4 vether1
10.0.0.1 00:00:00:00:00:01 UHLch 1 3 - 3 vether1
10.0.0.3 00:00:00:00:00:03 UHLl 0 0 - 1 vether1
10.0.0.255 10.0.0.3 UHb 0 0 - 1 vether1
# for if in bridge0 em2 vether{0,1}; do ifconfig $if; done
bridge0: flags=41<UP,RUNNING>
description: Bridge for external virtual NICs
index 9 llprio 3
groups: bridge
priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp
designated: id 00:00:00:00:00:00 priority 0
em2 flags=3<LEARNING,DISCOVER>
port 3 ifpriority 0 ifcost 0
vether0 flags=3<LEARNING,DISCOVER>
port 6 ifpriority 0 ifcost 0
vether1 flags=3<LEARNING,DISCOVER>
port 7 ifpriority 0 ifcost 0
Addresses (max cache: 100, timeout: 240):
00:00:00:00:00:01 em2 1 flags=0<>
em2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr xx:xx:xx:xx:xx:xx
description: External interface #1 (integrated NIC)
index 3 priority 0 llprio 3
media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
status: active
vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:00:00:00:02
description: Main external IP
index 6 priority 0 llprio 3
groups: vether egress
media: Ethernet autoselect
status: active
inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
vether1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
rdomain 1 mtu 1500
lladdr 00:00:00:00:00:03
description: Secondary external IP
index 7 priority 0 llprio 3
groups: vether
media: Ethernet autoselect
status: active
inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255
Footnotes:
[1] My ISP implements carrier-grade NAT, with an opt-out. If I supply a
MAC address, they will whitelist it and give it an external IP
number over DHCP. I can do this for more than one card but they must
have a unique MAC, meaning that I can not simply use an IP alias.
While experimenting with the configuration, I am using my own
internal DHCP server. The plan is to connect the OpenBSD server
directly to the ISP.
[2] It should not be necessary to have two vether, em2 should be able to
serve the purpose of the primary interface. However, I think I'm
hitting the bug described in the first paragraph here:
http://undeadly.org/cgi?action=article&sid=20160725144108
[3] The MAC addresses in this report are obfuscated: 10.0.0.xx maps to
00:00:00:00:00:xx to aid connecting a card with its IP. The virtual
interfaces has valid MAC addresses taken from retired NICs.
[4] http://search.cpan.org/dist/App-Asciio/lib/App/Asciio.pm
