Re: net.inet.ip.ifq.maxlen
http://www.undeadly.org/cgi?action=articlesid=20060927091645 is still mostly relevant. Great article. Thanks for the link and also for the other tips. Michel
Re: net.inet.ip.ifq.maxlen
Le 2012-09-04 13:52, Claudio Jeker a écrit : On Tue, Sep 04, 2012 at 10:16:41AM -0400, Michel Blais wrote: I've build a Xeon E3 with Intel i340 ethernet with 82580 chip. CPU is use up to 24% on the first core, congestion is now at 0.3/s. I still see drops in net.inet.ip.ifq.drops. 1131 drops in 81 hours. 1131 pkts dropped in 81hours or 291600 seconds. That is one packet every 4 minutes. That should not cause any trouble at all. Such drops can happen because the softnet interupt is unable to run for a long time. E.g. because of some magic SMM/NMI taking you CPU out for lunch. In which case you will get a massive burst once the CPU is back at work. I know this but the problem is that it always come in batch like 10 dropped paquets every 40 minutes so this is affecting voip and some player lost their game connectivity, etc. I'm now trying kern.pool_debug=1 but don't know where the output will go and can't find anything about the output. Will it be in dmesg or in a log ? What do you hope the get with enabling the pool corruption checker (apart from a slower machine)? Some answer mention it so I thinked it was a diagnostic tool that could be helpfull but maybe the wanted to make sure that I was disabled and that I didn't understand first. Also I would like to write again my rule but you like to know more about PF's ruleset optimization mechanisms. I see in pf.conf man page the following : Basic ruleset optimization does four things to improve the performance of ruleset evaluations: 1. remove duplicate rules 2. remove rules that are a subset of another rule 3. combine multiple rules into a table when advantageous 4. re-order the rules to improve evaluation performance I can handle 1, 2 and 3 fine without the optimisation but for the order of the rule, is there any doc on how to optimise the order of the rule order for best performance ? I was also not able to find anything about this. Rule evaluation happens in a certain order. The optimizer reorders the rules so that skip steps can do a better job (e.g. moving rules with the same source IP together). In general rules are evaluated in an order that is roughly interface, direction, rdomain, AF, proto, source IP and dst IP. If there is no match pf calculates something called skip steps which is a hint where to jump to in case of a failed match. In other words if all rules for em0 come first and the packet enters em1 we can skip all those rules after the first test. After the basic tests the proto is checked and additional per protocol evaluation happen (like TCP flags, ports, uid/gid checks...). These checks are not part of skip steps and are not influenced by the optimizer. Only the first 7 matter and are optimized. Perfect, that what I needed to know. That meen that I don't need this optimizer since I already do my own optimisation with anchor. Thanks -- Michel Blais Administrateur réseau / Network administrator Targo Communications www.targo.ca 514-448-0773
Re: net.inet.ip.ifq.maxlen
I've build a Xeon E3 with Intel i340 ethernet with 82580 chip. CPU is use up to 24% on the first core, congestion is now at 0.3/s. I still see drops in net.inet.ip.ifq.drops. 1131 drops in 81 hours. I'm now trying kern.pool_debug=1 but don't know where the output will go and can't find anything about the output. Will it be in dmesg or in a log ? Also I would like to write again my rule but you like to know more about PF's ruleset optimization mechanisms. I see in pf.conf man page the following : Basic ruleset optimization does four things to improve the performance of ruleset evaluations: 1. remove duplicate rules 2. remove rules that are a subset of another rule 3. combine multiple rules into a table when advantageous 4. re-order the rules to improve evaluation performance I can handle 1, 2 and 3 fine without the optimisation but for the order of the rule, is there any doc on how to optimise the order of the rule order for best performance ? I was also not able to find anything about this. Thanks Michel Le 2012-08-30 09:57, Michel Blais a écrit : Le 2012-08-30 08:59, Ryan McBride a écrit : On Wed, Aug 29, 2012 at 12:54:18PM -0400, Michel Blais wrote: How much can I increase net.inet.ip.ifq.maxlen ? I'm now at 2048 and still seeing increase in net.inet.ip.ifq.drops. This morning, it was at 21280 and now at 21328. A little bit of congestion increase is not the end of the world, but increasing the increasing the queue length will certainly increase your latency. Latency is fine (1 or 2 ms) but go up to 50 ms for maybe 40 or 50 maximized paquets ( paquet interval 0.01) and goes fine again. If paquets lost happen, look like it happen at the same time than this latency. I've change the système for a temporary more powerfull one (core 2 quad + 2 dual 82571EB) while I'm commanding and building new server and now the congestion have dropped from 3.9 to 0.8. More cores will not help; throwing more power at the problem may not be the solution, but if it is: the top performance will be a CPU with high clock speed (disable the other cores so that 'Turbo Boost' can crank the live core up), and the largest, fastest cache possible. You could also try setting kern.pool_debug=0. I know it was not the CPU since it was not even use at 50% and also know that pf can't take advantage of multiple core. I change to have ethernet card with better bus since I read that 82546 was limited with small paquet because of the PCI-X bus (no pci-e on the older xeon we use previously so add to change the whole machine). Source for limited 82546 : http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg03134.html I tryed with 2 dual 82571 card (PCI-E) like already writen and it result in lot less congestion (3.9 - 0.8). I now have my new servers (Xeon E3 with quad 82580 on PCI-E 2.0 bus) and will see if it's help. Anyway, I add to build new server since the core 2 quad was a desktop install temporary. Something I must specify, I use bi-nat to save public ip address and have thousand of bi-nat rule divided in some anchors. Thousands of rules is not a good idea if you can avoid it. This may be a little bit helped by your anchors, or the anchors may make it worse (PF's ruleset optimization mechanisms will not operate across anchors). Can you explain in more detail what you are doing with these bi-nat rules? -Ryan Since we can't have more IPv4 block from ARIN, we add to find a way to maximized those we already have. We then choose instead to use bi-nat to assign public address to our customer. Before we use bi-nat, it was a little less than 50% address lost in network, braodcast, gateway + not assing because we add to route 64 adresse for 30 or 40 customers, etc. -- Michel Blais Administrateur réseau / Network administrator Targo Communications www.targo.ca 514-448-0773
Re: net.inet.ip.ifq.maxlen
On Tue, Sep 04, 2012 at 10:16:41AM -0400, Michel Blais wrote: I've build a Xeon E3 with Intel i340 ethernet with 82580 chip. CPU is use up to 24% on the first core, congestion is now at 0.3/s. I still see drops in net.inet.ip.ifq.drops. 1131 drops in 81 hours. 1131 pkts dropped in 81hours or 291600 seconds. That is one packet every 4 minutes. That should not cause any trouble at all. Such drops can happen because the softnet interupt is unable to run for a long time. E.g. because of some magic SMM/NMI taking you CPU out for lunch. In which case you will get a massive burst once the CPU is back at work. I'm now trying kern.pool_debug=1 but don't know where the output will go and can't find anything about the output. Will it be in dmesg or in a log ? What do you hope the get with enabling the pool corruption checker (apart from a slower machine)? Also I would like to write again my rule but you like to know more about PF's ruleset optimization mechanisms. I see in pf.conf man page the following : Basic ruleset optimization does four things to improve the performance of ruleset evaluations: 1. remove duplicate rules 2. remove rules that are a subset of another rule 3. combine multiple rules into a table when advantageous 4. re-order the rules to improve evaluation performance I can handle 1, 2 and 3 fine without the optimisation but for the order of the rule, is there any doc on how to optimise the order of the rule order for best performance ? I was also not able to find anything about this. Rule evaluation happens in a certain order. The optimizer reorders the rules so that skip steps can do a better job (e.g. moving rules with the same source IP together). In general rules are evaluated in an order that is roughly interface, direction, rdomain, AF, proto, source IP and dst IP. If there is no match pf calculates something called skip steps which is a hint where to jump to in case of a failed match. In other words if all rules for em0 come first and the packet enters em1 we can skip all those rules after the first test. After the basic tests the proto is checked and additional per protocol evaluation happen (like TCP flags, ports, uid/gid checks...). These checks are not part of skip steps and are not influenced by the optimizer. Only the first 7 matter and are optimized. -- :wq Claudio
Re: net.inet.ip.ifq.maxlen
On 2012-09-04, Michel Blais mic...@targointernet.com wrote: I've build a Xeon E3 with Intel i340 ethernet with 82580 chip. CPU is use up to 24% on the first core, congestion is now at 0.3/s. First core..this implies MP. You might do better with UP as having the other cores unused may allow turbo boost to speed things up a bit more. I still see drops in net.inet.ip.ifq.drops. 1131 drops in 81 hours. I'm now trying kern.pool_debug=1 but don't know where the output will go and can't find anything about the output. Will it be in dmesg or in a log ? If you do 'show all pools' in ddb you'll get some extra messages if pool corruption was detected, I'm not sure if it shows up anywhere else. It has a big effect on performance, this is why it is only enabled for -current i.e. disabled for releases. Also I would like to write again my rule but you like to know more about PF's ruleset optimization mechanisms. I see in pf.conf man page the following : Basic ruleset optimization does four things to improve the performance of ruleset evaluations: 1. remove duplicate rules 2. remove rules that are a subset of another rule 3. combine multiple rules into a table when advantageous 4. re-order the rules to improve evaluation performance I can handle 1, 2 and 3 fine without the optimisation but for the order of the rule, is there any doc on how to optimise the order of the rule order for best performance ? I was also not able to find anything about this. http://www.undeadly.org/cgi?action=articlesid=20060927091645 is still mostly relevant.
Re: net.inet.ip.ifq.maxlen was WARNING: mclpools limit reached; increase kern.maxclusters and paquet lost
On Wed, Aug 29, 2012 at 12:54:18PM -0400, Michel Blais wrote: How much can I increase net.inet.ip.ifq.maxlen ? I'm now at 2048 and still seeing increase in net.inet.ip.ifq.drops. This morning, it was at 21280 and now at 21328. A little bit of congestion increase is not the end of the world, but increasing the increasing the queue length will certainly increase your latency. I've change the système for a temporary more powerfull one (core 2 quad + 2 dual 82571EB) while I'm commanding and building new server and now the congestion have dropped from 3.9 to 0.8. More cores will not help; throwing more power at the problem may not be the solution, but if it is: the top performance will be a CPU with high clock speed (disable the other cores so that 'Turbo Boost' can crank the live core up), and the largest, fastest cache possible. You could also try setting kern.pool_debug=0. Something I must specify, I use bi-nat to save public ip address and have thousand of bi-nat rule divided in some anchors. Thousands of rules is not a good idea if you can avoid it. This may be a little bit helped by your anchors, or the anchors may make it worse (PF's ruleset optimization mechanisms will not operate across anchors). Can you explain in more detail what you are doing with these bi-nat rules? -Ryan
Re: net.inet.ip.ifq.maxlen
Le 2012-08-30 08:59, Ryan McBride a écrit : On Wed, Aug 29, 2012 at 12:54:18PM -0400, Michel Blais wrote: How much can I increase net.inet.ip.ifq.maxlen ? I'm now at 2048 and still seeing increase in net.inet.ip.ifq.drops. This morning, it was at 21280 and now at 21328. A little bit of congestion increase is not the end of the world, but increasing the increasing the queue length will certainly increase your latency. Latency is fine (1 or 2 ms) but go up to 50 ms for maybe 40 or 50 maximized paquets ( paquet interval 0.01) and goes fine again. If paquets lost happen, look like it happen at the same time than this latency. I've change the système for a temporary more powerfull one (core 2 quad + 2 dual 82571EB) while I'm commanding and building new server and now the congestion have dropped from 3.9 to 0.8. More cores will not help; throwing more power at the problem may not be the solution, but if it is: the top performance will be a CPU with high clock speed (disable the other cores so that 'Turbo Boost' can crank the live core up), and the largest, fastest cache possible. You could also try setting kern.pool_debug=0. I know it was not the CPU since it was not even use at 50% and also know that pf can't take advantage of multiple core. I change to have ethernet card with better bus since I read that 82546 was limited with small paquet because of the PCI-X bus (no pci-e on the older xeon we use previously so add to change the whole machine). Source for limited 82546 : http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg03134.html I tryed with 2 dual 82571 card (PCI-E) like already writen and it result in lot less congestion (3.9 - 0.8). I now have my new servers (Xeon E3 with quad 82580 on PCI-E 2.0 bus) and will see if it's help. Anyway, I add to build new server since the core 2 quad was a desktop install temporary. Something I must specify, I use bi-nat to save public ip address and have thousand of bi-nat rule divided in some anchors. Thousands of rules is not a good idea if you can avoid it. This may be a little bit helped by your anchors, or the anchors may make it worse (PF's ruleset optimization mechanisms will not operate across anchors). Can you explain in more detail what you are doing with these bi-nat rules? -Ryan Since we can't have more IPv4 block from ARIN, we add to find a way to maximized those we already have. We then choose instead to use bi-nat to assign public address to our customer. Before we use bi-nat, it was a little less than 50% address lost in network, braodcast, gateway + not assing because we add to route 64 adresse for 30 or 40 customers, etc. -- Michel Blais Administrateur réseau / Network administrator Targo Communications www.targo.ca 514-448-0773
Re: net.inet.ip.ifq.maxlen and altq's qlimit
* Daniel Melameth [EMAIL PROTECTED] [2007-07-09 21:49]: I'm not certain how they interrelate, but if one is experiencing congestion, and, as a result, tweaks net.inet.ip.ifq.maxlen to compensate, is safe to assume that if altq is in use on the same system qlimit should match or be less than the value of net.inet.ip.ifq.maxlen? How does one determine the optimal qlimit value? ifq.maxlenand altq qlimits are not directly related. the formeris the length of the global ip intr queue, the latter the max for each of the numerous output queues. there is no easy way to determine the optimal qlimit value, it is always a compromise, andmost of the time, doesn't matter all that much -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg Amsterdam
