OK, so following Luigi's suggestion we've re-enabled AIM, set max_interrupt_rate to 8000 (matching igb), and reduced number of queues to 6. We'll have next peak in about 14 hours, I'll try to capture and record history of the per-queue interrupt rate. It still remains somewhat puzzling why something that worked with old-gen hardware/driver almost out of the box now needs some extra tuning. Thanks everyone for useful suggestions in any case!
On Tue, Aug 11, 2015 at 4:04 PM, Luigi Rizzo <ri...@iet.unipi.it> wrote: > On Wed, Aug 12, 2015 at 12:46 AM, Maxim Sobolev <sobo...@sippysoft.com> > wrote: > > Hi Luigi, thanks for the input. Are you sure about the number of queues? > > Keep in mind we are not using netmap or anything as advanced, so we need > lot > > of parallel pipes just to push packet through the whole UDP/IP kernel > stack. > > To make things worse, our app cannot use connected sockets at the > moment, so > > we are having worst case scenario wrt kernel overhead. Single thread > sender > > on that hardware is limited to only 225K PPS until it hits CPU limit. We > use > > 8 queues on other older servers with I350 NICs. My send-only testing > shows > > that the hardware peaks at about 5 concurrent threads for connected > sockets > > and 7 concurrent threads for unconnected. This is with default 8 hardware > > queues. The test is somewhat crippled since it only tests outgoing > traffic. > > > > thing is, you are using 8 queues per NIC so the total number of different > interrupts is huge and will likely eat all of your cores. > This is why i'd try to start low and ramp up later. > > > http://bit.ly/1MiLFHg > > > > With regards to the moderation, what effect it has on the traffic? Is NIC > > going to drop packets once it starts moderating interrupts? > > moderation only means you get larger batches on each interrupt, > which overall makes the system more efficient. The NIC will only drop > packets when the CPU is not fast enough to drain them,. > > (this said, it might be a good idea to try netmap, passing packets > that you do not know how to handle to the host stack, and instead > intercepting those that you can deal with in netmap. those > unconnected udp sockets are the ideal case for netmap.) > > cheers > luigi > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
