One of the things not readily evident in trying to scale up, is the cost of even the most basic routing table lookup. A lot of good work in this area landed in linux 4.1 and 4.2 (see a couple posts here: https://vincent.bernat.im/en/blog/2017-performance-progression-ipv4-route-lookup-linux )
Lookup time for even the smallest number of routes is absolutely miserable for IPv6 - https://vincent.bernat.im/en/blog/2017-ipv6-route-lookup-linux I think one of the biggest driving factors of the whole TSO/GRO thing is due to trying to get smaller packets through this phase of the kernel, and not that they are so much more efficient at the card itself. Given the kerfuffle over here ( https://github.com/systemd/systemd/issues/9725 ) I'd actually like to come up with a way to move the linux application socket buffers to the post-lookup side of the routing table. We spend a lot of extra time bloating up superpackets just so they are cheaper to route. TCAMs are expensive as hell, but the addition of even a small one, readily accessible to userspace or from the kernel, might help in the general case. I've actually oft wished to be able to offload these sort of lookups into higher level algorithms and languages like python, as a general purpose facility. Hey, if we can have giant GPUs, why can't our cpus have tcams? programmable TCAM support got enabled in a recent (mellonox?) product. Can't find the link at the moment TCAMs of course, is where big fat dedicated routers and switches shine, over linux - and even arp table lookups are expensive in linux, though I'm not sure if anyone has looked lately. _______________________________________________ Bloat mailing list [email protected] https://lists.bufferbloat.net/listinfo/bloat
