Dear list,
We have a performance issue impacting all our infrastructure behind our
OpenBSD: two front BGP/CARP routers with 1Gb/s transit. It seams to
occur since we have upgraded to 7.5, both of the servers are up to date.
A simple ssh (without login) on our the router have notable performance
issue:
* Usual operational state
router$ top -s 1
CPU00: 0.2% user, 0.0% nice, 1.0% sys, 0.8% spin, 24.7% intr, 73.3% idle
CPU01: 3.6% user, 0.0% nice, 20.1% sys, 1.4% spin, 0.0% intr, 74.9% idle
CPU02: 4.7% user, 0.0% nice, 15.9% sys, 1.4% spin, 0.0% intr, 78.0% idle
CPU03: 0.0% user, 0.0% nice, 3.0% sys, 1.0% spin, 0.0% intr, 96.0% idle
CPU04: 0.0% user, 0.0% nice, 0.8% sys, 0.0% spin, 0.0% intr, 99.2% idle
CPU05: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU06: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU07: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU08: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU09: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU10: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU11: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
observer# ping -i 0.1 router
[…]
64 bytes from router: icmp_seq=122 ttl=255 time=0.242 ms
64 bytes from router: icmp_seq=123 ttl=255 time=0.232 ms
64 bytes from router: icmp_seq=124 ttl=255 time=0.257 ms
64 bytes from router: icmp_seq=125 ttl=255 time=0.316 ms
64 bytes from router: icmp_seq=126 ttl=255 time=0.244 ms
64 bytes from router: icmp_seq=127 ttl=255 time=0.265 ms
64 bytes from router: icmp_seq=128 ttl=255 time=0.333 ms
64 bytes from router: icmp_seq=129 ttl=255 time=0.239 ms
64 bytes from router: icmp_seq=130 ttl=255 time=0.338 ms
64 bytes from router: icmp_seq=131 ttl=255 time=0.375 ms
64 bytes from router: icmp_seq=132 ttl=255 time=0.400 ms
64 bytes from router: icmp_seq=133 ttl=255 time=0.275 ms
* On ssh connection (even with no auth key):
joe$ ssh foo@router
foo@router: Permission denied (publickey).
router$ top -s 1
CPU00: 0.0% user, 0.0% nice, 5.8% sys, 1.9% spin, 65.0% intr, 27.2% idle
CPU01: 0.0% user, 0.0% nice, 12.6% sys, 16.5% spin, 0.0% intr, 70.9% idle
CPU02: 1.9% user, 0.0% nice, 11.7% sys, 36.9% spin, 0.0% intr, 49.5% idle
CPU03: 0.0% user, 0.0% nice, 1.9% sys, 1.0% spin, 0.0% intr, 97.1% idle
CPU04: 0.0% user, 0.0% nice, 0.0% sys, 1.0% spin, 0.0% intr, 99.0% idle
CPU05: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU06: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU07: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU08: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU09: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU10: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
CPU11: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle
cpu0 goes 65% intr here, quite often we can see 100%
and the spin time skyrockets
observer# ping -i 0.1 router
[…]
64 bytes from router: icmp_seq=524 ttl=255 time=0.242 ms
64 bytes from router: icmp_seq=525 ttl=255 time=0.360 ms
64 bytes from router: icmp_seq=526 ttl=255 time=0.279 ms
64 bytes from router: icmp_seq=527 ttl=255 time=568.051 ms
64 bytes from router: icmp_seq=528 ttl=255 time=468.094 ms
64 bytes from router: icmp_seq=529 ttl=255 time=433.769 ms
64 bytes from router: icmp_seq=530 ttl=255 time=334.208 ms
64 bytes from router: icmp_seq=531 ttl=255 time=234.224 ms
64 bytes from router: icmp_seq=532 ttl=255 time=134.255 ms
64 bytes from router: icmp_seq=533 ttl=255 time=55.565 ms
64 bytes from router: icmp_seq=534 ttl=255 time=36.819 ms
64 bytes from router: icmp_seq=535 ttl=255 time=0.316 ms
64 bytes from router: icmp_seq=536 ttl=255 time=0.292 ms
64 bytes from router: icmp_seq=537 ttl=255 time=0.320 ms
64 bytes from router: icmp_seq=538 ttl=255 time=0.302 ms
Sacha
