On Sun, Dec 26, 2021 at 07:46:01AM +0000, Simon Baker wrote:
> Hi,
>
> Struggling a bit debugging something, and hoping someone can point me in the
> right direction.
ok. after staring at this for a while im pretty sure it's an actual bug
rather than a misconfiguration.
> I???ve got 4 physical intel nics, all configured as part of a veb bridge.
> The veb bridge itself has two vports attached, one with an address and one
> without:
>
> cat /etc/hostname.vport0
> inet 172.16.0.250 255.255.255.0
> group trusted
> up
>
> cat /etc/hostname.vport1
> group vlan-interface
> link0
as an aside, link0 on a vport doesn't do anything.
> up
>
> The hostname.veb0 file contains this:
> add em0
> add em1
> add em2
> add em3
> add vport0
> add vport1
> link0
> up
>
> This setup is working fine for all hosts on my main LAN, and everything is as
> expected. However I???ve tried and (partially) failed in adding some
> VLAN???s to the veb.
>
> For example, here???s one of the vlan configurations:
> cat /etc/hostname.vlan210
> inet 172.16.210.2 255.255.255.0 172.16.210.255
> parent vport1
> vlan 210
> description "VLAN 210 - A/V & Media Devices???
> up
>
> Note the following only discusses one VLAN, but the issue is present on all
> of the configured VLANs.
>
> From a host on the VLAN network, it can connect outbound to the internet
> absolutely fine - but it cannot talk back to the main network. Strangely,
> running tcpdump on interfaces shows traffic moving as (possibly) expected -
> but packets never seem to appear on the wire to the downstream host.
>
> In the following example, Volumio is a host on the VLAN 210 as above,
> attempting to send an ICMP echo request to a host on the main lan. First up,
> here???s a PF log showing the permitted packet:
>
> Dec 25 20:41:13.342006 rule 86/(match) pass out on vport0: 172.16.210.13 >
> 172.16.0.1: icmp: echo request
>
> (Note, I still get the same issues even with disabling pf)
>
> Next, here???s the packet on the vport1 interface from above:
>
> 20:41:22.663129 dc:a6:32:4d:9a:4c fe:e1:ba:d3:54:a5 8100 102: 802.1Q vid 210
> pri 1 volumio.av.kaizo.lan > nas.kaizo.lan: icmp: echo request (DF)
>
> Now, here???s the packet on the vport0 interface:
> 20:41:22.663145 fe:e1:ba:d2:e4:93 68:05:ca:4a:7c:18 ip 98:
> volumio.av.kaizo.lan > nas.kaizo.lan: icmp: echo request
>
> However, this is where it stops. I see no matching packet on the veb0
> interface, nor do I see a packet egress on the physical em1 interface, to
> which the host ???nas??? is connected to. Obviously I don???t see the packet
> on that host, either.
>
> I???m a little perplexed as to what???s going on here - it???s almost as if
> the veb doesn???t believe it???s responsible for this packet. It seems to be
> happily routing packets from the LAN to hosts on a VLAN, it???s just the
> return traffic that never arrives.
you're right, the veb doesn't think it should handle the packet.
veb sets and clears a flag on packets going in and out of vport
interfaces as a sort of loop prevention mechanism. because vlan packets
are handled before veb can clear this flag, the packet ends up being
marked as inside veb when it goes through the network stack. when it
comes out the stack on a vport interface again, it gets dropped because
of this flag still being set.
there's a diff below that moves away from the flag to try and avoid this
problem. can you give it a go in your setup?
alternatively, i think you could use a separate veb per vlan, but
that's a lot of boilerplate...
> For completeness, below are output of ifconfig for the interfaces (edited).
>
> Simon.
>
> veb0: flags=9943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,LINK0,MULTICAST>
> index 12 llprio 3
> groups: veb
> em0 flags=3<LEARNING,DISCOVER>
> port 1 ifpriority 0 ifcost 0
> em1 flags=3<LEARNING,DISCOVER>
> port 2 ifpriority 0 ifcost 0
> em2 flags=3<LEARNING,DISCOVER>
> port 3 ifpriority 0 ifcost 0
> em3 flags=3<LEARNING,DISCOVER>
> port 4 ifpriority 0 ifcost 0
> vport0 flags=3<LEARNING,DISCOVER>
> port 19 ifpriority 0 ifcost 0
> vport1 flags=3<LEARNING,DISCOVER>
> port 20 ifpriority 0 ifcost 0
> Addresses (max cache: 100, timeout: 240):
> ???snip???.
> 68:05:ca:4a:7c:18 em1 0 flags=0<>
> ???.snip???.
> fe:e1:ba:d2:e4:93 vport0 0 flags=0<>
> fe:e1:ba:d3:54:a5 vport1 0 flags=0<>
>
> vport0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> lladdr fe:e1:ba:d2:e4:93
> index 19 priority 0 llprio 3
> groups: vport trusted
> inet 172.xx.xx.250 netmask 0xffffff00 broadcast 172.16.0.255
>
> vport1: flags=9943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,LINK0,MULTICAST> mtu
> 1500
> lladdr fe:e1:ba:d3:54:a5
> index 20 priority 0 llprio 3
> groups: vport vlan-interface
>
> vlan210: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> lladdr fe:e1:ba:d3:54:a5
> description: VLAN 210 - A/V & Media Devices
> index 16 priority 0 llprio 3
> encap: vnetid 210 parent vport1 txprio packet rxprio outer
> groups: vlan
> inet 172.16.210.2 netmask 0xffffff00 broadcast 172.16.210.255
Index: if_veb.c
===================================================================
RCS file: /cvs/src/sys/net/if_veb.c,v
retrieving revision 1.21
diff -u -p -r1.21 if_veb.c
--- if_veb.c 8 Nov 2021 04:15:46 -0000 1.21
+++ if_veb.c 27 Dec 2021 22:20:30 -0000
@@ -117,6 +117,8 @@ struct veb_port {
struct ifnet *p_ifp0;
struct refcnt p_refs;
+ int (*p_enqueue)(struct ifnet *, struct mbuf *);
+
int (*p_ioctl)(struct ifnet *, u_long, caddr_t);
int (*p_output)(struct ifnet *, struct mbuf *, struct sockaddr *,
struct rtentry *);
@@ -233,6 +235,8 @@ struct vport_softc {
unsigned int sc_dead;
};
+static int vport_if_enqueue(struct ifnet *, struct mbuf *);
+
static int vport_ioctl(struct ifnet *, u_long, caddr_t);
static int vport_enqueue(struct ifnet *, struct mbuf *);
static void vport_start(struct ifqueue *);
@@ -901,7 +905,7 @@ veb_broadcast(struct veb_softc *sc, stru
continue;
}
- if_enqueue(ifp0, m); /* XXX count error? */
+ (*tp->p_enqueue)(ifp0, m); /* XXX count error */
}
smr_read_leave();
@@ -946,7 +950,7 @@ veb_transmit(struct veb_softc *sc, struc
counters_pkt(ifp->if_counters, ifc_opackets, ifc_obytes,
m->m_pkthdr.len);
- if_enqueue(ifp0, m); /* XXX count error? */
+ (*tp->p_enqueue)(ifp0, m); /* XXX count error */
return (NULL);
drop:
@@ -955,6 +959,12 @@ drop:
}
static struct mbuf *
+veb_vport_input(struct ifnet *ifp0, struct mbuf *m, uint64_t dst, void *brport)
+{
+ return (m);
+}
+
+static struct mbuf *
veb_port_input(struct ifnet *ifp0, struct mbuf *m, uint64_t dst, void *brport)
{
struct veb_port *p = brport;
@@ -966,11 +976,6 @@ veb_port_input(struct ifnet *ifp0, struc
caddr_t if_bpf;
#endif
- if (ISSET(m->m_flags, M_PROTO1)) {
- CLR(m->m_flags, M_PROTO1);
- return (m);
- }
-
if (!ISSET(ifp->if_flags, IFF_RUNNING))
return (m);
@@ -1059,7 +1064,6 @@ veb_port_input(struct ifnet *ifp0, struc
etherbridge_map(&sc->sc_eb, p, src);
CLR(m->m_flags, M_BCAST|M_MCAST);
- SET(m->m_flags, M_PROTO1);
if (!ETH64_IS_MULTICAST(dst)) {
struct veb_port *tp = NULL;
@@ -1245,6 +1249,7 @@ veb_add_port(struct veb_softc *sc, const
struct ifnet *ifp0;
struct veb_ports *port_list;
struct veb_port *p;
+ int isvport;
int error;
NET_ASSERT_LOCKED();
@@ -1263,6 +1268,8 @@ veb_add_port(struct veb_softc *sc, const
goto put;
}
+ isvport = (ifp0->if_enqueue == vport_enqueue);
+
error = ether_brport_isset(ifp0);
if (error != 0)
goto put;
@@ -1283,12 +1290,18 @@ veb_add_port(struct veb_softc *sc, const
SMR_TAILQ_INIT(&p->p_vr_list[0]);
SMR_TAILQ_INIT(&p->p_vr_list[1]);
+ p->p_enqueue = isvport ? vport_if_enqueue : if_enqueue;
p->p_ioctl = ifp0->if_ioctl;
p->p_output = ifp0->if_output;
if (span) {
port_list = &sc->sc_spans;
+ if (isvport) {
+ error = EPROTONOSUPPORT;
+ goto free;
+ }
+
p->p_brport.eb_input = veb_span_input;
p->p_bif_flags = IFBIF_SPAN;
} else {
@@ -1299,7 +1312,8 @@ veb_add_port(struct veb_softc *sc, const
goto free;
p->p_bif_flags = IFBIF_LEARNING | IFBIF_DISCOVER;
- p->p_brport.eb_input = veb_port_input;
+ p->p_brport.eb_input = isvport ?
+ veb_vport_input : veb_port_input;
}
p->p_brport.eb_port_take = veb_eb_brport_take;
@@ -1323,7 +1337,7 @@ veb_add_port(struct veb_softc *sc, const
port_list->l_count++;
ether_brport_set(ifp0, &p->p_brport);
- if (ifp0->if_enqueue != vport_enqueue) { /* vport is special */
+ if (!isvport) { /* vport is special */
ifp0->if_ioctl = veb_p_ioctl;
ifp0->if_output = veb_p_output;
}
@@ -2176,6 +2190,20 @@ vport_down(struct vport_softc *sc)
}
static int
+vport_if_enqueue(struct ifnet *ifp, struct mbuf *m)
+{
+ /*
+ * switching an l2 packet toward a vport means pushing it
+ * into the network stack. this function exists to make
+ * if_vinput compat with veb calling if_enqueue.
+ */
+
+ if_vinput(ifp, m);
+
+ return (0);
+}
+
+static int
vport_enqueue(struct ifnet *ifp, struct mbuf *m)
{
struct arpcom *ac;
@@ -2185,24 +2213,19 @@ vport_enqueue(struct ifnet *ifp, struct
caddr_t if_bpf;
#endif
+ /*
+ * a packet sent from the l3 stack out a vport goes into
+ * veb for switching out another port.
+ */
+
#if NPF > 0
/*
- * the packet is about to leave the l3 stack and go into
- * the l2 switching space, or it's coming from a switch space
- * into the network stack. either way, there's no relationship
- * between pf states in those different places.
+ * there's no relationship between pf states in the l3 stack
+ * and the l2 bridge.
*/
pf_pkt_addr_changed(m);
#endif
- if (ISSET(m->m_flags, M_PROTO1)) {
- /* packet is coming from a bridge */
- if_vinput(ifp, m);
- return (0);
- }
-
- /* packet is going to the bridge */
-
ac = (struct arpcom *)ifp;
smr_read_enter();
@@ -2211,6 +2234,8 @@ vport_enqueue(struct ifnet *ifp, struct
eb->eb_port_take(eb->eb_port);
smr_read_leave();
if (eb != NULL) {
+ struct mbuf *(*input)(struct ifnet *, struct mbuf *,
+ uint64_t, void *) = eb->eb_input;
struct ether_header *eh;
uint64_t dst;
@@ -2225,7 +2250,10 @@ vport_enqueue(struct ifnet *ifp, struct
eh = mtod(m, struct ether_header *);
dst = ether_addr_to_e64((struct ether_addr *)eh->ether_dhost);
- m = (*eb->eb_input)(ifp, m, dst, eb->eb_port);
+
+ if (input == veb_vport_input)
+ input = veb_port_input;
+ m = (*input)(ifp, m, dst, eb->eb_port);
error = 0;
