Hey, having a dual-node setup of 6.0 in prod, I decided to move forward with one of machines and upgrade to 6.1-stable. Ending up in benchmark tool ”locking” the 6.1 machine.
Background: Nodes are Xeon E5-2642v3 3.4Ghz x12, 16G RAM, 64G DOM modules as hdd, 4x X540T (ix) - 2x on-board and 2x PCI-card. All 4x X540T are connected to 2x Cisco Nexus 3000-series, creating an LACP trunk (1x on-board + 1x PCI). trunk0 - external (VLAN), 1x NIC connected to switch1 and 1x NIC connected to switch2 (ix0 + ix3) trunk1 - internal (VLAN) , 1x NIC connected to switch1 and 1x NIC connected to switch2 (ix1 + ix2) As I have 2x Nexus 3000, VPC is configured and sitting on top of LACP trunk on their end. Each obsd node have several carp interfaces configured on top of trunk0. Only one carp interface on trunk1 - carp1. Each switch acting as a default gw (VRRP configured) for any existing VLAN, except one towards trunk1. Default gateway for those switches is IP on carp1. Those switches run OSPF as well as obsd nodes do. obsd nodes are the front line, facing the Internet. (2x uplink goes into 2x Nexus and then traffic is passed to 2x obsd.) Running relayd with SSL-offload and plain HTTP. Except relayd, there is ospfd, ntpd, snmpd, and bgpd(for distributed blacklisting around other global nodes). The problem: While doing a bench with https://github.com/wg/wrk <https://github.com/wg/wrk> from my laptop (OS X, 1Gbps max. pipe) agains the environment (HTTPS) relayd experienced problems with handling the traffic. shell# ./wrk -t16 -c1500 -d90s —latency <https://URL> wrk hammering apache 2.4(behind those nodes), serving a txt file with avg 7k-10k req/s as an output: wrk -t16 -c1500 -d90s --latency https:/<URL>/ping.txt Running 2m test @ https://<URL>/ping.txt 16 threads and 1500 connections Thread Stats Avg Stdev Max +/- Stdev Latency 131.17ms 70.91ms 1.97s 91.70% Req/Sec 651.06 135.80 1.09k 84.95% Latency Distribution 50% 131.90ms 75% 144.63ms 90% 159.63ms 99% 230.92ms 927039 requests in 1.50m, 190.12MB read Socket errors: connect 0, read 0, write 0, timeout 1330 Requests/sec: 10290.54 Transfer/sec: 2.11MB wrk hammering apache 2.4, mod_proxy_balance, with NodeJS nodes behind apache: wrk -t16 -c1500 -d90s --latency https://<URL>/nodejs Running 2m test @ https://<URL>/nodejs 16 threads and 1500 connections Thread Stats Avg Stdev Max +/- Stdev Latency 445.91ms 518.66ms 2.00s 83.49% Req/Sec 56.80 26.89 180.00 68.48% Latency Distribution 50% 217.57ms 75% 374.15ms 90% 1.50s 99% 1.95s 80673 requests in 1.50m, 1.12GB read Socket errors: connect 0, read 5534, write 0, timeout 18099 Requests/sec: 895.42 Transfer/sec: 12.72MB ’top’ showed none interrupting at all, but rather heavy system load values and some user values. 20-30% - user 80-90% - system relayd (12 forks as the number of cores) - 99% usage. I basically killed both machines running 6.0, thus my decision to upgrade to 6.1. However, during the tests against 6.0, my ssh session never got terminated (”kicked out”) even with this hight load (0% CPU idle). 6.1 showed different symptoms - ssh session termination, login via web based IPMI GUI hanging after log in part, ping not responding(from the switches and node1 which is 6.0 yet). After a while, with bench aborted, 6.1 eventually let me in via ssh (terminal via IPMI stil hanging). snmpd which been running (remember), been polled by other sys doing graphs. What been seen on those graphs is high rate of output err pkts on trunks, not NICs (ix) them selves. Also, syslog, with enabled ’log all’ for relayd showed a lot of ’buffer timeout event’, ospfd yeilding about ’no buffer space available’. I had to modd relayd.conf to spawn only 8 preforks instead of 12 and kern.maxclusters=24576 #12288 kern.maxfiles=65536 #32768 in order to survive the bench (e.g.. having ssh session alive). Values commented out are from the 6.0 setup. I’m looking for any advice here, which hopefully will lead to a stable and performant setup. Configuration follows. ———sysct.conf (obsd 6.0)———— net.inet.ip.forwarding=1 net.inet.ipcomp.enable=1 # 1=Enable the IPCOMP protocol net.inet.etherip.allow=1 # 1=Enable the Ethernet-over-IP protocol net.inet.tcp.ecn=1 # 1=Enable the TCP ECN extension net.inet.carp.preempt=1 # 1=Enable carp(4) preemption net.inet.carp.log=3 # log level of carp(4) info, default 2 ddb.panic=0 # 0=Do not drop into ddb on a kernel panic ddb.console=1 # 1=Permit entry of ddb from the console kern.pool_debug=0 net.inet.ip.maxqueue=2048 kern.somaxconn=4096 kern.maxclusters=12288 kern.maxfiles=32768 net.inet.ip.ifq.maxlen=2048 ————login.conf——————— relayd:\ :maxproc-max=31:\ :openfiles-cur=65536:\ :openfiles-max=65536:\ :tc=daemon: —————pf.conf——————— set block-policy drop set limit { states 3000000, frags 2000, src-nodes 1000000 } —————relayd.conf——————— interval 10 timeout 1000 prefork 8 #12 log all ——>>>>>>>> for debuging the situation shell# netstat -m 1227 mbufs in use: 626 mbufs allocated to data 189 mbufs allocated to packet headers 412 mbufs allocated to socket names and addresses 0/232/64 mbuf 2048 byte clusters in use (current/peak/max) 423/2865/120 mbuf 2112 byte clusters in use (current/peak/max) 0/160/64 mbuf 4096 byte clusters in use (current/peak/max) 0/200/64 mbuf 8192 byte clusters in use (current/peak/max) 0/14/112 mbuf 9216 byte clusters in use (current/peak/max) 0/20/80 mbuf 12288 byte clusters in use (current/peak/max) 0/16/64 mbuf 16384 byte clusters in use (current/peak/max) 0/8/64 mbuf 65536 byte clusters in use (current/peak/max) 23400 Kbytes allocated to network (5% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Kernel is stock, latest via syspatch. P.S. Ifq.drops been never observed, nor high ifq.len (max. 5 pkts in queue) PF had max. 290k states. Br //mxb
