Re: [pfSense] High-latency when traffic reaches 80% wirespeed

[Ivo Tonev]

run "top -SH" to find the top cpu consuming tasks


On Thu, Oct 5, 2017 at 8:44 AM, Christoph Haas 
wrote:

> Am Mittwoch, den 04.10.2017, 15:05 -0400 schrieb ED Fochler:
> > I have a similar situation and I solved it with limiters.  I'm also a
> fan of limiters to ensure fair sharing of uplink bandwidth by internal
> users.  I haven't tried changing system tunables though, so that solution
> may be better.
>
> So far the situation was better this morning. But the web interface
> became unresponsive and the OpenVPN daemon died. So I'm still scared.
>
> >
> Nothing is sent through the limiter until you create a rule that catches
> the traffic and routes it through the limiter, so you're not going to
> accidentally slow everything down just by creating a rule.
>
> I will try that.
>
> >
> The behavior you're speaking of sounds like your machine is getting maxed
> out by interrupts or some internal bandwidth.  Setting up a limiter sounds
> like a better solution than pushing the hardware to the point of unrefined
> behavior.
>
> Yes, I suspect something like that, too. The system load is going up
> heavily (Load >=5) sometimes. However the web interface claims that the
> load is around 30%. RAM and state tables look fine, too.
>
> On Linux-based systems I regularly use iptables rules and often go near
> wire speed. But the system load rarely goes up noticably. So I wonder
> what part is really causing that load.
>
> I ran "top" this morning and saw that the "filterlog" process was at
> the top of the list. My firewall rules though do not do any logging at
> the moment. Could that still be a problem?
>
> Thanks for your suggestions so far. I'll try them all.
>
> …Christoph
>
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold
>



-- 
Ivo R. Tonev
+55 61 98409-2642
i...@tonev.com.br
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] High-latency when traffic reaches 80% wirespeed

[Christoph Haas]

Am Mittwoch, den 04.10.2017, 15:05 -0400 schrieb ED Fochler:
> I have a similar situation and I solved it with limiters.  I'm also a fan of 
> limiters to ensure fair sharing of uplink bandwidth by internal users.  I 
> haven't tried changing system tunables though, so that solution may be better.

So far the situation was better this morning. But the web interface
became unresponsive and the OpenVPN daemon died. So I'm still scared.

> 
Nothing is sent through the limiter until you create a rule that catches the 
traffic and routes it through the limiter, so you're not going to accidentally 
slow everything down just by creating a rule.

I will try that.

> 
The behavior you're speaking of sounds like your machine is getting maxed out 
by interrupts or some internal bandwidth.  Setting up a limiter sounds like a 
better solution than pushing the hardware to the point of unrefined behavior.

Yes, I suspect something like that, too. The system load is going up
heavily (Load >=5) sometimes. However the web interface claims that the
load is around 30%. RAM and state tables look fine, too.

On Linux-based systems I regularly use iptables rules and often go near
wire speed. But the system load rarely goes up noticably. So I wonder
what part is really causing that load.

I ran "top" this morning and saw that the "filterlog" process was at
the top of the list. My firewall rules though do not do any logging at
the moment. Could that still be a problem?

Thanks for your suggestions so far. I'll try them all.

…Christoph

___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] High-latency when traffic reaches 80% wirespeed

[Christoph Haas]

Am Mittwoch, den 04.10.2017, 19:13 + schrieb Steve Yates:
> Christoph, if you are using CARP/HA for your two routers, see 
> https://redmine.pfsense.org/issues/4310 "Limiters + HA results in hangs on 
> secondary."

Not yet but I'll look out to that. Thanks.

> Alternatively if the overnight traffic is due to an rsync, rsync can limit 
> its own bandwidth also.

I suspect some kind of backup job. There are many different data
transfers going on in the network. So a general solution like limiting
sounds better.

___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] High-latency when traffic reaches 80% wirespeed

[ED Fochler]

I have a similar situation and I solved it with limiters.  I'm also a fan of 
limiters to ensure fair sharing of uplink bandwidth by internal users.  I 
haven't tried changing system tunables though, so that solution may be better.

Nothing is sent through the limiter until you create a rule that catches the 
traffic and routes it through the limiter, so you're not going to accidentally 
slow everything down just by creating a rule.

The behavior you're speaking of sounds like your machine is getting maxed out 
by interrupts or some internal bandwidth.  Setting up a limiter sounds like a 
better solution than pushing the hardware to the point of unrefined behavior.

ED.


> On 2017, Oct 4, at 4:08 AM, Christoph Haas  wrote:
> 
> Dear list,
> 
> I have become a huge fan of pfSense and managed to replace our old
> routers at work by two nifty Netgate SG-4860 gateways. They work nearly
> perfectly. I just have a few seperate internal VLANs (e.g. for
> administration, monitoring and backup) that give me a headache. Every
> day at the same time(s) there are spikes in traffic (I can see in the
> dashboard) between two VLANs. Traffic goes up to pretty much 800 Mbps
> for 1-2 minutes.
> 
> During that time our monitoring system goes wild. High latencies and
> even ping losses. CPU load of the router is shown at around 50%. Once
> the traffic goes below 800 Mbps all is instantly fine again.
> 
> I tried to simplify the firewall rules (e.g. let through all the
> traffic) but that did not help. Is there anything I can do? Any hidden
> switches? Anything to find and fix the situation? Traffic shaping for
> ICMP? Unicorn dust?
> 
> Thanks in advance for your hints.
> 
> …Christoph
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold

___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] High-latency when traffic reaches 80% wirespeed

[Ivo Tonev]

You can try rise some "System tunables"


net.inet.tcp.recvspace 524288
net.inet.tcp.sendspace 524288
net.raw.recvspace 524288
net.inet.raw.recvspace 524288
net.raw.sendspace 524288
net.inet.raw.maxdgram 524288
net.link.ifqmaxlen 2048
net.inet.tcp.recvbuf_inc 65536
net.inet.udp.recvspace 524288
net.inet.tcp.sendbuf_inc 65536
net.inet.tcp.mssdflt 1460
net.inet.tcp.minmss 536

On Wed, Oct 4, 2017 at 5:08 AM, Christoph Haas 
wrote:

> Dear list,
>
> I have become a huge fan of pfSense and managed to replace our old
> routers at work by two nifty Netgate SG-4860 gateways. They work nearly
> perfectly. I just have a few seperate internal VLANs (e.g. for
> administration, monitoring and backup) that give me a headache. Every
> day at the same time(s) there are spikes in traffic (I can see in the
> dashboard) between two VLANs. Traffic goes up to pretty much 800 Mbps
> for 1-2 minutes.
>
> During that time our monitoring system goes wild. High latencies and
> even ping losses. CPU load of the router is shown at around 50%. Once
> the traffic goes below 800 Mbps all is instantly fine again.
>
> I tried to simplify the firewall rules (e.g. let through all the
> traffic) but that did not help. Is there anything I can do? Any hidden
> switches? Anything to find and fix the situation? Traffic shaping for
> ICMP? Unicorn dust?
>
> Thanks in advance for your hints.
>
> …Christoph
> ___
> pfSense mailing list
> https://lists.pfsense.org/mailman/listinfo/list
> Support the project with Gold! https://pfsense.org/gold




-- 
Ivo R. Tonev
+55 61 98409-2642
i...@tonev.com.br
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[pfSense] High-latency when traffic reaches 80% wirespeed

[Christoph Haas]

Dear list,

I have become a huge fan of pfSense and managed to replace our old
routers at work by two nifty Netgate SG-4860 gateways. They work nearly
perfectly. I just have a few seperate internal VLANs (e.g. for
administration, monitoring and backup) that give me a headache. Every
day at the same time(s) there are spikes in traffic (I can see in the
dashboard) between two VLANs. Traffic goes up to pretty much 800 Mbps
for 1-2 minutes.

During that time our monitoring system goes wild. High latencies and
even ping losses. CPU load of the router is shown at around 50%. Once
the traffic goes below 800 Mbps all is instantly fine again.

I tried to simplify the firewall rules (e.g. let through all the
traffic) but that did not help. Is there anything I can do? Any hidden
switches? Anything to find and fix the situation? Traffic shaping for
ICMP? Unicorn dust?

Thanks in advance for your hints.

…Christoph
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold