On Tue, Jul 7, 2015 at 2:31 PM, Willy Tarreau <w...@1wt.eu> wrote: Hi Willy,
Thank you once again for the quick response, and apologize for my tardiness. Unless I'm wrong, for me ixgbe does its own RSS and uses the Rx queues. Each > Rx queue is bound to an IRQ, and each IRQ may be delivered to a set of > CPUs. > I'd say that I've never seen even a iota of CPU usage on the wrong CPU when > using ixgbe, so I'm quite sure that some of your IRQs are not properly > bound, > which could be verified by simply running "grep eth" in /proc/interrupts, > though I can easily undertand that it's not easy to read with 48 columns The irq counts increase for all cores though haproxy runs only on even cores. And smp_affinity is set correctly, going from 1, 2, 4, 8, 10, 20, 40, 80, 100 (all in hex), etc. I have not seen this issue under lower load, but when cpu's are at 100%, I am getting this behavior. I will try to recreate with low and high load and validate this statement. No it's not that small because a flow is something ephemeral. Even when > you run with hundreds of thousands of concurrent connections, the time > between an outgoing ACK and the next data packet is short enough for the > flow to be matched, especially on a local network when it's a matter of > hundreds of microseconds at worst. > Yes, that seems reasonable. But if the 8K flows are exhausted during a single tx iteration, I assume RX for some flows will not find the flow cached for RSS. Just out of curiosity, why not configure the lab to reproduce the same > setup as the prod, ie 12 processes as well ? > Yes, I will try this in a little while, as we are busy with production setup at present. > At the very least, check the interrupts per CPU. There I'm sure you'll > find that you missed some of them and that they're still delivered to > the wrong socket. Also keep in mind that a down-up cycle on an interface > unbinds the IRQs, so you have to rebind them by hand. This is something > which easily happens by mistake during troubleshooting sessions. > Since we are doing this for hundreds of systems, the IRQ's are being set at boot time through a rc2 script. Thanks once again for your inputs. Regards, - Krishna Kumar -- ------------------------------------------------------------------------------------------------------------------------------------------ This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Although Flipkart has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments
