I brought up packet batching in today's meeting, and mentioned Mellanox asking about a feature like xmit_more in Linux.
To recap, "xmit_more" is a flag to an skb in linux that the stack can use to tell the driver that there are more packets coming immediately, and so it is allowed to delay writing any doorbells to notify the NIC about the transmission. This offers a fairly large amount of savings, and it avoids mmio access to the NIC doorbells, and (in Mellanox's case) can be used to reduce transmit completions. There is a description in more detail of how it works in linux at http://netoptimizer.blogspot.com/2014/10/unlocked-10gbps-tx-wirespeed-smallest.html (as I said .. benchmarks .. ) It looks like they are bulking up things as they come out of their qdisc layer, above the drivers. Note that they have a fairly sophisticated set of qdiscs that are usable with modern, multi-queue drivers, as well as centralized software queuing that all drivers use. Without properly adding all those layers, the simplest thing to do would be to just have tcp_output (and other proto output routines) set a similar flag when they are looping, and sending down mbufs to ip_output. The problem with that is simply that (at least for our internet facing workloads at netflix), sending more data on a socket than the tso max seg size is quite rare. (see appended dtrace output from a machine serving ~90K connections at ~85Gb/s) What is unclear to me is whether or not linux would see any benefit from xmit more in our sort of workload. My guess is that it would not, as the qdiscs will not be delaying things, and they'll still be mostly limited to client ack packing, so will also rarely be sending more than a TSO max seg size. Drew c094.ord001.dev# dtrace -s ./xmit_len.d dtrace: script './xmit_len.d' matched 1 probe ^C 0 value ------------- Distribution ------------- count < 1514 |@@@ 1454289 1514 |@@@@@@@@ 4601582 2514 |@@@@@@ 3529106 3514 |@@@@@ 2902881 4514 | 150322 5514 |@@@ 1433990 6514 |@ 722206 7514 | 121788 8514 |@ 855718 9514 |@ 441330 10514 | 86424 11514 |@ 742827 12514 |@ 405603 13514 | 33753 14514 |@ 432492 15514 |@ 451240 16514 |@ 309429 17514 | 67458 18514 | 243994 19514 | 275286 20514 | 27206 21514 | 198855 22514 |@ 291526 23514 | 20636 24514 | 210524 25514 | 175945 26514 | 13526 27514 | 156973 28514 | 174066 29514 | 115377 30514 | 24194 31514 | 138833 32514 | 104053 33514 | 23385 34514 | 131241 35514 | 92529 36514 | 27125 37514 | 88839 38514 | 76915 39514 | 6554 40514 | 86519 41514 | 70078 42514 | 55289 43514 | 16175 44514 | 74943 45514 | 66529 46514 | 12766 47514 | 66808 48514 | 51397 49514 | 14020 50514 | 46155 51514 | 37579 52514 | 9177 53514 | 39342 54514 | 31170 55514 | 8080 56514 | 34876 57514 | 40574 58514 | 32921 59514 | 6768 60514 | 30252 61514 | 25051 62514 | 5609 63514 | 174059 >= 64514 |@ 445810 0 65516 c094.ord001.dev# cat xmit_len.d fbt::mlx5e_xmit:entry { m = (struct mbuf *)arg1; /* @len[0] = quantize(m->M_dat.MH.MH_pkthdr.len); */ @len[0] = lquantize(m->M_dat.MH.MH_pkthdr.len, 1514, 65000, 1000); @m[0] = max(m->M_dat.MH.MH_pkthdr.len); } _______________________________________________ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-transport To unsubscribe, send any mail to "[email protected]"
