Hi Piotr, Please see below. Raja
anyone elaborate regarding what meaning following conuters and what is the relation to each other and to HOLD queue. - ds_flow_queue_limit_exceeded > Each flow can hold up to 3 packets while waiting for the agent to resolve the > flow. Any subsequent packets are dropped if the flow has still not been > resolved by the agent and this counter is incremented. - ds_flow_unusable > This is most likely due to the limit on the number of HOLD next hops being > reached. - flows.exceptions > I’m not sure which counter this one corresponds to. Perhaps someone else can > comment. How I can monitor number of packets in Hold queue and how often I reach to itslimit ? > When the limit on the number of HOLD next hops is reached, the flow_unusable > counter is incremented. >I don’t think there is a single counter to monitor the number of held packets. >Each flow has stats and we >could add the stats of all HOLD flows to arrive at this. It seems that one of my problem may be udp services like dns and logging via udp. > We are planning to allow certain UDP ports to be configured to allow flow > handling to be more efficient. This will be in the next release. They cause vrouter to create new flow for each packet. Already I have lowered value for flow age time. Here on specific vrouter where is udp receiver I can see large spikes of values in ds_flow_unusable like 10k or more. However at the same time the counter ds_flow_queue_limit_exceeded reads small values . > It depends on how many packets are received for each flow. If it is DNS, it > is unlikely to cause flow queue limit to be exceeded as we have room for 3 packets. On other vrouters with mainly tcp services I can see only rather small values in ds_flow_queue_limit_exceeded. > This is expected as the SYN will result in the flow getting created and more > packets are no received as the SYN ACK is pending. But I'm not seeing any values in ds_flow_unusable counter. Does it mean that vrouter may drops packets also in this case? > It probably means that you are not getting a huge burst of connection > requests. Can anyone share knowledge how to tune vrouter parameters to achieve better performance. > I suggest that you try the 2.22/3.0 release to see if the changes mentioned > above improve the performance. We have also made changes to age flows based on TCP state in addition to improving the flow setup rate. On 20 October 2015 at 11:25, Édouard Thuleau <[email protected]<mailto:[email protected]>> wrote: Hi Anand, I'm giving you my test results. The strange behavior I found is when the limit is reached, the flow rate completely decrease to zero. Do you have any idea why I saw that? Regards, Édouard. On Thu, Oct 15, 2015 at 1:58 PM, Édouard Thuleau <[email protected]<mailto:[email protected]>> wrote: Hi Anand, I'm happy to heard your work on that. By next release you mean the 2.22 or the 3.0? Do you have a bug or blueprint associated to that work? And any design documentations on how you will solve that? Did you measure the same results that mines? Just to be sure I configured correctly my vrouters and I don't loose any flow/s on my platform. Regards, Édouard. On Thu, Oct 15, 2015 at 10:48 AM, Anand H Krishnan <[email protected]<mailto:[email protected]>> wrote: Hello Edouard, We are improving the flow setup rate and you should see a vastly improved rate in the next release. Thanks, Anand ________________________________ From: Dev <[email protected]<mailto:[email protected]>> on behalf of Édouard Thuleau <[email protected]<mailto:[email protected]>> Sent: Thursday, October 15, 2015 1:34 PM To: Rajagopalan Sivaramakrishnan Cc: [email protected]<mailto:[email protected]> Subject: Re: [opencontrail-dev] Flow unsusable over new 1000req/s Hi Raja, Thanks for that explanation. So with that mechanism, the vrouter is not able to manage more than 1k new flow per seconds? It's not very high, modern HTTP server can handle 500k req/s on a bare metal server [1] (that seems be done on the loopback interface of the web server, not through a real network, no NIC. It's more around 50k from an external node). Did you measure the same result with the vrouter? Do you know a way to improve that? [1] https://lowlatencyweb.wordpress.com/2012/03/20/500000-requestssec-modern-http-servers-are-fast/ [https://lowlatencyweb.files.wordpress.com/2012/03/nginx1.png]<https://lowlatencyweb.wordpress.com/2012/03/20/500000-requestssec-modern-http-servers-are-fast/> 500,000 requests/sec – Modern HTTP servers are fast | The ... A modern HTTP server running on somewhat recent hardware is capable of servicing a huge number of requests with very low latency. Here's a plot showing ... Read more...<https://lowlatencyweb.wordpress.com/2012/03/20/500000-requestssec-modern-http-servers-are-fast/> Édouard. On Thu, Oct 15, 2015 at 7:11 AM, Rajagopalan Sivaramakrishnan <[email protected]<mailto:[email protected]>> wrote: Hi Edouard, When a new flow is created, the first packet of the flow is sent to the vrouter agent to lookup policy. While the flow is for the agent to resolve it, it is marked as a “HOLD" flow. Vrouter limits the number of such flows to 4096. Packets for any new flows are dropped by vrouter (and the “Flow Unusable” counter is incremented) when this limit has been crossed. This should be an ephemeral condition and new flows will be created once the agent has resolved the older flows. Raja From: Dev <[email protected]<mailto:[email protected]>> on behalf of Édouard Thuleau <[email protected]<mailto:[email protected]>> Date: Wednesday, October 14, 2015 at 9:42 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [opencontrail-dev] Flow unsusable over new 1000req/s Hi, I'm doing some network stress tests on Contrail. One of them is to measure at which rate a vrouter can create new ephemeral TCP sessions. To do that test, I'm using a tsung generator distributed on different VM (five) and a nginx server that simply return an HTTP 204 code. I setted a floating IP on the nginx server and tsung clients use it as HTTP endpoint. I also had to tune some system and nginx parameters. On the Contrail side, I just tested it on 1.10 branch for the moment. The kernel module is loaded with 'vr_flow_entries' setted to 2097152 and the vrouter agent 'FLOWS.max_vm_flows' to 20%. The result is the vrouter starts to drop flow with reason "Flow Unusable" when the tsung client try to send 1000 HTTP requests per seconds. Do you think my results are correct? Do you see any mistake in my test bed? Someone already tried that type of test? If yes, what's your results? Regards, Édouard. _______________________________________________ Dev mailing list [email protected]<mailto:[email protected]> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
_______________________________________________ Dev mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
