Re: [pfSense Support] CARP switchover to backup because of high traffic
Evgeny Yurchenko wrote: Jim Pingle wrote: Evgeny Yurchenko wrote: Yesterday it happened twice on one of my production firewalls. CPU load was less than 10%. Did not pay attention at the moment but accoring to RRD number of states was not unusual - 4-5k. I reproduced it in my lab - only test connection, so number of states was less than 100. When this happens, check the output of ifconfig -a on the master when it won't take back over, see what advskew it is advertising. There are certain failure states that cause it to set an advskew of 240 regardless of what it is actually configured to be. Figuring out what caused that, however, can be a bit trickier. I push quite a lot of traffic through my pfSense boxes and have never seen them failover in this manner. Nightly backups push just about wire speed through my CARP pair (100MBit). Agian hit the same situation on production firewall. All carp interfaces show carp: BACKUP vhid xxx advbase 1 advskew 0 like this: carp0: flags=49UP,LOOPBACK,RUNNING mtu 1500 inet 10.0.0.244 netmask 0xff00 carp: BACKUP vhid 100 advbase 1 advskew 0 On all interfaces see only partner's packets like this # tcpdump -ni vlan1 vrrp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vlan1, link-type EN10MB (Ethernet), capture size 96 bytes 19:11:39.871724 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:41.264295 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:42.656753 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:44.049203 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:45.441655 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:46.834109 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 ^C # sysctl net.inet.ip.intr_queue_drops net.inet.ip.intr_queue_drops: 0 but now there is no load. If anybody can give any advice I can keep this situation for some time as it is afterbusiness hours Friday. Thanks, Evgeny. One more time on different pfSense cluster. If I pay for support would somebody be able to login and see what is going on here? Thanks. Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
On Fri, Nov 13, 2009 at 4:31 PM, Evgeny Yurchenko evg.yu...@rogers.com wrote: If I pay for support would somebody be able to login and see what is going on here? Sure, absolutely. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
Chris Buechler wrote: On Fri, Nov 13, 2009 at 4:31 PM, Evgeny Yurchenko evg.yu...@rogers.com wrote: If I pay for support would somebody be able to login and see what is going on here? Sure, absolutely. BTW https://portal.pfsense.org/index.php/subscribe-for-access does not look nice in IE. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
Chris Buechler wrote: On Fri, Nov 13, 2009 at 4:31 PM, Evgeny Yurchenko evg.yu...@rogers.com wrote: If I pay for support would somebody be able to login and see what is going on here? Sure, absolutely. Paid. Should we proceed off list? Thanks. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
Jim Pingle wrote: Evgeny Yurchenko wrote: Yesterday it happened twice on one of my production firewalls. CPU load was less than 10%. Did not pay attention at the moment but accoring to RRD number of states was not unusual - 4-5k. I reproduced it in my lab - only test connection, so number of states was less than 100. When this happens, check the output of ifconfig -a on the master when it won't take back over, see what advskew it is advertising. There are certain failure states that cause it to set an advskew of 240 regardless of what it is actually configured to be. Figuring out what caused that, however, can be a bit trickier. I push quite a lot of traffic through my pfSense boxes and have never seen them failover in this manner. Nightly backups push just about wire speed through my CARP pair (100MBit). Agian hit the same situation on production firewall. All carp interfaces show carp: BACKUP vhid xxx advbase 1 advskew 0 like this: carp0: flags=49UP,LOOPBACK,RUNNING mtu 1500 inet 10.0.0.244 netmask 0xff00 carp: BACKUP vhid 100 advbase 1 advskew 0 On all interfaces see only partner's packets like this # tcpdump -ni vlan1 vrrp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vlan1, link-type EN10MB (Ethernet), capture size 96 bytes 19:11:39.871724 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:41.264295 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:42.656753 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:44.049203 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:45.441655 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 19:11:46.834109 IP 10.0.0.243 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype none, intvl 1s, length 36 ^C # sysctl net.inet.ip.intr_queue_drops net.inet.ip.intr_queue_drops: 0 but now there is no load. If anybody can give any advice I can keep this situation for some time as it is afterbusiness hours Friday. Thanks, Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
On 07/10/09 18:47, Evgeny Yurchenko wrote: Has anybody noticed this behavior? The simplest set up: two pfSenses with LAN WAN and CARP on both interfaces (with separate interface for SYNC). When there is little traffic active pfSense sends CARP packets with priority 0 every second, everything is ok. Gradually increasing traffic you reach the point when active pfSense starts sending CARP packets not regularily: 1.5, 2, 3 seconds and finally stops sending them completely. Of course at this point backup pfSense kicks in. When you remove traffic former active pfSense does not restore its active role (does not any CARP packets). what's the CPU load at that time, and how full is the state table? - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
Paul Mansfield wrote: On 07/10/09 18:47, Evgeny Yurchenko wrote: Has anybody noticed this behavior? The simplest set up: two pfSenses with LAN WAN and CARP on both interfaces (with separate interface for SYNC). When there is little traffic active pfSense sends CARP packets with priority 0 every second, everything is ok. Gradually increasing traffic you reach the point when active pfSense starts sending CARP packets not regularily: 1.5, 2, 3 seconds and finally stops sending them completely. Of course at this point backup pfSense kicks in. When you remove traffic former active pfSense does not restore its active role (does not any CARP packets). what's the CPU load at that time, and how full is the state table? Yesterday it happened twice on one of my production firewalls. CPU load was less than 10%. Did not pay attention at the moment but accoring to RRD number of states was not unusual - 4-5k. I reproduced it in my lab - only test connection, so number of states was less than 100. Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
On Thu, Oct 8, 2009 at 11:24 AM, Evgeny Yurchenko evg.yu...@rogers.com wrote: Yesterday it happened twice on one of my production firewalls. CPU load was less than 10%. Did not pay attention at the moment but accoring to RRD number of states was not unusual - 4-5k. I reproduced it in my lab - only test connection, so number of states was less than 100. Evgeny. I would lean toward hardware. We regularly push 20 megabit out one of my CARP clusters and I do not see this behavior. If something is preempting the network stack (CARP) from sending its Heartbeats than it's doing what it is designed to do. Probably not what you want to hear but I would look at the hardware closer, interrupts, etc. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
Scott Ullrich wrote: On Thu, Oct 8, 2009 at 11:24 AM, Evgeny Yurchenko evg.yu...@rogers.com wrote: Yesterday it happened twice on one of my production firewalls. CPU load was less than 10%. Did not pay attention at the moment but accoring to RRD number of states was not unusual - 4-5k. I reproduced it in my lab - only test connection, so number of states was less than 100. Evgeny. I would lean toward hardware. We regularly push 20 megabit out one of my CARP clusters and I do not see this behavior. If something is preempting the network stack (CARP) from sending its Heartbeats than it's doing what it is designed to do. Probably not what you want to hear but I would look at the hardware closer, interrupts, etc. Scott Thanks I will. 20 Mbit/s is nothing though... Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
On Thu, Oct 8, 2009 at 11:42 AM, Evgeny Yurchenko evg.yu...@rogers.com wrote: Thanks I will. 20 Mbit/s is nothing though... I agree but you failed to mention how much traffic you are pushing. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
Scott Ullrich wrote: On Thu, Oct 8, 2009 at 11:42 AM, Evgeny Yurchenko evg.yu...@rogers.com wrote: Thanks I will. 20 Mbit/s is nothing though... I agree but you failed to mention how much traffic you are pushing. Scott Yes, sorry. It was about 100Mb/s - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
On Thu, Oct 8, 2009 at 12:51 PM, Evgeny Yurchenko evg.yu...@rogers.com wrote: Yes, sorry. It was about 100Mb/s During heavy load what does this sysctl show? sysctl net.inet.ip.intr_queue_drops Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] CARP switchover to backup because of high traffic
Evgeny Yurchenko wrote: Yesterday it happened twice on one of my production firewalls. CPU load was less than 10%. Did not pay attention at the moment but accoring to RRD number of states was not unusual - 4-5k. I reproduced it in my lab - only test connection, so number of states was less than 100. When this happens, check the output of ifconfig -a on the master when it won't take back over, see what advskew it is advertising. There are certain failure states that cause it to set an advskew of 240 regardless of what it is actually configured to be. Figuring out what caused that, however, can be a bit trickier. I push quite a lot of traffic through my pfSense boxes and have never seen them failover in this manner. Nightly backups push just about wire speed through my CARP pair (100MBit). - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
[pfSense Support] CARP switchover to backup because of high traffic
Has anybody noticed this behavior? The simplest set up: two pfSenses with LAN WAN and CARP on both interfaces (with separate interface for SYNC). When there is little traffic active pfSense sends CARP packets with priority 0 every second, everything is ok. Gradually increasing traffic you reach the point when active pfSense starts sending CARP packets not regularily: 1.5, 2, 3 seconds and finally stops sending them completely. Of course at this point backup pfSense kicks in. When you remove traffic former active pfSense does not restore its active role (does not any CARP packets). Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
