Hi Misc, I'm making some trials and benchmarks about pf's divert. My test environment is like this;
I have 2 Linux devices and I have an OpenBSD device which are directly connected to an OpenBSD Device. This OpenBSD device acts likes a router. Network settings in OpenBSD: --------- vertigo# ifconfig em1 em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:30:18:0a:a6:2e index 2 priority 0 llprio 3 media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause) status: active inet 172.20.35.1 netmask 0xffffff00 broadcast 172.20.35.255 vertigo# ifconfig em2 em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:30:18:0a:a6:2f index 3 priority 0 llprio 3 media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause) status: active inet 172.20.36.1 netmask 0xffffff00 broadcast 172.20.36.255 vertigo# ------- ------- vertigo# sysctl net.inet.ip.forwarding net.inet.ip.forwarding=1 ------- my pf.conf is like this; ----------- vertigo# cat /etc/pf.conf # $OpenBSD: pf.conf,v 1.55 2017/12/03 20:40:04 sthen Exp $ # # See pf.conf(5) and /etc/examples/pf.conf set skip on lo # em1 pass in log quick on em1 from em1:network to172.20.36.0/24 pass out log quick on em1 from172.20.36.0/24to em1:network # em2 pass in log quick on em2 from em2:network to172.20.35.0/24 pass out log quick on em2 from172.20.35.0/24to em2:network # block all block log quick all block return # block stateless traffic pass # establish keep-state # By default, do not permit remote connections to X11 block return in on ! lo0 proto tcp to port 6000:6010 # Port build user does not need network block return out log proto {tcp udp} user _pbuild -------- On Linux-1, I'm making benchmark tests to Linux-2 with qperf, -------- vertigo@linux1:~$ qperf 172.20.35.2 tcp_bw tcp_lat -t 10 tcp_bw: bw = 115 MB/sec tcp_lat: latency = 157 us vertigo@linux1:~$ qperf 172.20.35.2 udp_bw udp_lat -t 10 udp_bw: send_bw = 120 MB/sec recv_bw = 120 MB/sec udp_lat: latency = 158 us vertigo@linux1:~$ --------- After that, I updated the pf.conf and diverted packages to port 700 ------ vertigo# cat /etc/pf.conf # $OpenBSD: pf.conf,v 1.55 2017/12/03 20:40:04 sthen Exp $ # # See pf.conf(5) and /etc/examples/pf.conf set skip on lo # em1 pass in log quick on em1 from em1:network to172.20.36.0/24 pass out log quick on em1 from172.20.36.0/24to em1:network divert-packet port 700 # em2 pass in log quick on em2 from em2:network to172.20.35.0/24 pass out log quick on em2 from172.20.35.0/24to em2:network divert-packet port 700 # block all block log quick all block return # block stateless traffic pass # establish keep-state # By default, do not permit remote connections to X11 block return in on ! lo0 proto tcp to port 6000:6010 # Port build user does not need network block return out log proto {tcp udp} user _pbuild ------ On port 700, I run the example program on OpenBSD divert manpage. https://man.openbsd.org/divert.4 ----- #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <netinet/ip.h> #include <netinet/tcp.h> #include <arpa/inet.h> #include <stdio.h> #include <string.h> #include <err.h> #define DIVERT_PORT 700 int main(int argc, char *argv[]) { int fd, s; struct sockaddr_in sin; socklen_t sin_len; fd = socket(AF_INET, SOCK_RAW, IPPROTO_DIVERT); if (fd == -1) err(1, "socket"); memset(&sin, 0, sizeof(sin)); sin.sin_family = AF_INET; sin.sin_port = htons(DIVERT_PORT); sin.sin_addr.s_addr = 0; sin_len = sizeof(struct sockaddr_in); s = bind(fd, (struct sockaddr *) &sin, sin_len); if (s == -1) err(1, "bind"); for (;;) { ssize_t n; char packet[IP_MAXPACKET]; struct ip *ip; struct tcphdr *th; int hlen; char src[48], dst[48]; memset(packet, 0, sizeof(packet)); n = recvfrom(fd, packet, sizeof(packet), 0, (struct sockaddr *) &sin, &sin_len); if (n == -1) { warn("recvfrom"); continue; } if (n < sizeof(struct ip)) { warnx("packet is too short"); continue; } ip = (struct ip *) packet; hlen = ip->ip_hl << 2; if (hlen < sizeof(struct ip) || ntohs(ip->ip_len) < hlen || n < ntohs(ip->ip_len)) { warnx("invalid IPv4 packet"); continue; } th = (struct tcphdr *) (packet + hlen); if (inet_ntop(AF_INET, &ip->ip_src, src, sizeof(src)) == NULL) (void)strlcpy(src, "?", sizeof(src)); if (inet_ntop(AF_INET, &ip->ip_dst, dst, sizeof(dst)) == NULL) (void)strlcpy(dst, "?", sizeof(dst)); printf("%s:%u -> %s:%u\n", src, ntohs(th->th_sport), dst, ntohs(th->th_dport) ); n = sendto(fd, packet, n, 0, (struct sockaddr *) &sin, sin_len); if (n == -1) warn("sendto"); } return 0; } And I made the same test again; There is a big difference in TCP benchmark tests, I repeated the tests, but the results did not change. --------- vertigo@linux1:~$ qperf 172.20.35.2 tcp_bw tcp_lat -t 10 tcp_bw: bw = 32.6 KB/sec tcp_lat: latency = 214 us vertigo@linux1:~$ qperf 172.20.35.2 udp_bw udp_lat -t 10 udp_bw: send_bw = 120 MB/sec recv_bw = 111 MB/sec udp_lat: latency = 214 us vertigo@linux1:~$ ---------- I could not understand why this big difference occurs. Is there any way to improve "divert" s performance? Sent with [ProtonMail](https://protonmail.com) Secure Email.