Re: [pfSense Support] Inbound load balancer performance under heavy load.
we've also had problems with inbound load balancing which we thought was just crappy ISP - a small number of http connections would quietly fail, or take a very long time and then fail, so we ended up using haproxy instead; this on 1.2-release as well as 1.2.2. (note to people: please trim replies when quoting and turn off HTML, this thread has become unreadable due to bad quoting and horrible HTML styles). - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] Inbound load balancer performance under heavy load.
On Fri, Jun 26, 2009 at 11:25 AM, Scott Ullrichsullr...@gmail.com wrote: On Fri, Jun 26, 2009 at 8:07 AM, Paul Mansfieldit-admin-pfse...@taptu.com wrote: we've also had problems with inbound load balancing which we thought was just crappy ISP - a small number of http connections would quietly fail, or take a very long time and then fail, so we ended up using haproxy instead; this on 1.2-release as well as 1.2.2. (note to people: please trim replies when quoting and turn off HTML, this thread has become unreadable due to bad quoting and horrible HTML styles). If you get a chance, please grab a 2.0 snapshot ISO and test our new haproxy package (it is for 2.0 only right now). Please e-mail me off list if you need help getting this going. I just remembered its a rcs branch but it would be easy to get going. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
RE: [pfSense Support] Inbound load balancer performance under heavy load.
Thank for your response, however the limitations on the featured list are not the cause of the problem. I am happy with the load balancer to equally distribute the load, also happy with the firewall not checking for a valid response. but there seems to be any other limitation not listed. Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:11 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. _ From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 10:29:03 +0100 Subject: [pfSense Support] Inbound load balancer performance under heavy load. Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time. We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before. We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds. So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine. so the only other thing that could be causing the problem was pfSense. however I couldn't find any indication of anything going wrong but the traffic graph. memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc. After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend. and that fixed the problem of requests not getting through and the traffic graph was again stable. I wonder if it is there any known issue with the inbound load balancer. I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~21, as when load testing we tested from a bunch of ~10 IPs. The problem is that we do need load balancing, mainly for redundancy of our systems at the back end. The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we don't know if pfSense is actually able to take the load. Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONET file:///D:/LABELS-PARTNERS/Playstation/Emails/email.jpg http://gfx2.hotmail.com/mail/w3/ltr/i_safe.gif The contents of this e-mail and any attachments/inserts are strictly confidential and sent for the attention of the addressee/s only. This e-mail might contain confidential and/or privileged material therefore if you are not the addressee/s, any distribution, review, disclosure, copying or other use of this e-mail and any attachments/inserts is prohibited without written provision. If received in error, please advise the sender and delete it from your system immediately. Statements and opinions expressed may not represent those of Rants Ltd and any representations or commitments in this email are subject to contract. Rants Ltd (trading as VidZone) --- Hi. From features list: Inbound Load Balancing Inbound load balancing is used to distribute load between multiple servers. This is commonly used with web servers, mail servers, and others. Servers that fail to respond to ping requests or TCP port connections are removed from the pool. Limitations * Equally distributes load between all available servers - unable to unequally distribute load between servers at this time. * Only checks if the server responds to pings or TCP port connections. Cannot check if the server is returning valid content. More info on: http://pfsense.org/index.php?option=com_contenttask=viewid=40Itemid=43 Cheers. Tebano. _ check out the rest of the Windows LiveT. More than mail-Windows LiveT goes way beyond your inbox. More than http://www.microsoft.com/windows/windowslive/ messages
RE: [pfSense Support] Inbound load balancer performance under heavy load.
From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 11:33:54 +0100 Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. Thank for your response, however the limitations on the featured list are not the cause of the problem… I am happy with the load balancer to equally distribute the load, also happy with the firewall not checking for a valid response… but there seems to be any other limitation not listed… Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA - Hi Josè. Sorry for the misunderstanding. I was simply trying to say that there aren't other limitation (that I know), except the ones I've pasted You. Probably my answer wasn't really wroten correctly... sorry!Cheers.Tebano. From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:11 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 10:29:03 +0100 Subject: [pfSense Support] Inbound load balancer performance under heavy load. Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time… We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before… We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds… So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine… so the only other thing that could be causing the problem was pfSense… however I couldn’t find any indication of anything going wrong but the traffic graph… memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc… After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend… and that fixed the problem of requests not getting through and the traffic graph was again stable… I wonder if it is there any known issue with the inbound load balancer… I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~21, as when load testing we tested from a bunch of ~10 IPs… The problem is that we do need load balancing, mainly for redundancy of our systems at the back end… The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we don’t know if pfSense is actually able to take the load… Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONE™ The contents of this e-mail and any attachments/inserts are strictly confidential and sent for the attention of the addressee/s only. This e-mail might contain confidential and/or privileged material therefore if you are not the addressee/s, any distribution, review, disclosure, copying or other use of this e-mail and any attachments/inserts is prohibited without written provision. If received in error, please advise the sender and delete it from your system immediately. Statements and opinions expressed may not represent those of Rants Ltd and any representations or commitments in this email are subject to contract. Rants Ltd (trading as VidZone) --- Hi. From features list: Inbound Load Balancing Inbound load balancing is used to distribute load between multiple servers. This is commonly used with web servers, mail servers, and others. Servers that fail to respond to ping requests or TCP port connections are removed from the pool. Limitations Equally distributes load between all available servers - unable to unequally distribute load between servers at this time. Only checks
RE: [pfSense Support] Inbound load balancer performance under heavy load.
It is fine Tebano, I appreciate your answer and as you said there are not other limitations documented nor any other issues I could find anywhere, and I did spend some time researching Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:44 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. _ From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 11:33:54 +0100 Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. Thank for your response, however the limitations on the featured list are not the cause of the problem I am happy with the load balancer to equally distribute the load, also happy with the firewall not checking for a valid response but there seems to be any other limitation not listed Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA - Hi Josè. Sorry for the misunderstanding. I was simply trying to say that there aren't other limitation (that I know), except the ones I've pasted You. Probably my answer wasn't really wroten correctly... sorry! Cheers. Tebano. From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:11 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. _ From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 10:29:03 +0100 Subject: [pfSense Support] Inbound load balancer performance under heavy load. Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine so the only other thing that could be causing the problem was pfSense however I couldnt find any indication of anything going wrong but the traffic graph memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend and that fixed the problem of requests not getting through and the traffic graph was again stable I wonder if it is there any known issue with the inbound load balancer I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~21, as when load testing we tested from a bunch of ~10 IPs The problem is that we do need load balancing, mainly for redundancy of our systems at the back end The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we dont know if pfSense is actually able to take the load Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONE file:///D:/LABELS-PARTNERS/Playstation/Emails/email.jpg http://gfx2.hotmail.com/mail/w3/ltr/i_safe.gif The contents of this e-mail and any attachments/inserts are strictly confidential and sent for the attention of the addressee/s only. This e-mail might contain confidential and/or privileged material therefore if you are not the addressee/s, any distribution, review, disclosure, copying or other use of this e-mail and any attachments/inserts is prohibited without written provision. If received in error, please advise the sender and delete it from your system immediately. Statements and opinions expressed may not represent those of Rants Ltd and any representations
Re: [pfSense Support] Inbound load balancer performance under heavy load.
Can you please try a later snapshot after 11062009 it seems you had problems with sticky-connections! Though without analysis i might be compeletly WRONG. Ermal On Fri, Jun 12, 2009 at 12:52 PM, Jose Hernandezj...@vidzone.tv wrote: It is fine Tebano, I appreciate your answer and as you said there are not other limitations documented nor any other issues I could find anywhere, and I did spend some time researching… Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:44 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 11:33:54 +0100 Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. Thank for your response, however the limitations on the featured list are not the cause of the problem… I am happy with the load balancer to equally distribute the load, also happy with the firewall not checking for a valid response… but there seems to be any other limitation not listed… Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA - Hi Josè. Sorry for the misunderstanding. I was simply trying to say that there aren't other limitation (that I know), except the ones I've pasted You. Probably my answer wasn't really wroten correctly... sorry! Cheers. Tebano. From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:11 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 10:29:03 +0100 Subject: [pfSense Support] Inbound load balancer performance under heavy load. Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time… We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before… We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds… So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine… so the only other thing that could be causing the problem was pfSense… however I couldn’t find any indication of anything going wrong but the traffic graph… memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc… After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend… and that fixed the problem of requests not getting through and the traffic graph was again stable… I wonder if it is there any known issue with the inbound load balancer… I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~21, as when load testing we tested from a bunch of ~10 IPs… The problem is that we do need load balancing, mainly for redundancy of our systems at the back end… The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we don’t know if pfSense is actually able to take the load… Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONE™ The contents of this e-mail and any attachments/inserts are strictly confidential and sent for the attention of the addressee/s only. This e-mail might contain confidential and/or privileged material therefore if you are not the addressee/s, any distribution, review, disclosure, copying or other use of this e-mail and any attachments
RE: [pfSense Support] Inbound load balancer performance under heavy load.
From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 11:52:13 +0100 Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. It is fine Tebano, I appreciate your answer and as you said there are not other limitations documented nor any other issues I could find anywhere, and I did spend some time researching… Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA In every case, if You've the possibility, I always suggest an update to last release.Moreover, because a new one is arriving, so I'm sure there's a difference between every-ones...:-)Cheers.Tebano. From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:44 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 11:33:54 +0100 Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. Thank for your response, however the limitations on the featured list are not the cause of the problem… I am happy with the load balancer to equally distribute the load, also happy with the firewall not checking for a valid response… but there seems to be any other limitation not listed… Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA - Hi Josè. Sorry for the misunderstanding. I was simply trying to say that there aren't other limitation (that I know), except the ones I've pasted You. Probably my answer wasn't really wroten correctly... sorry! Cheers. Tebano. From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:11 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 10:29:03 +0100 Subject: [pfSense Support] Inbound load balancer performance under heavy load. Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time… We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before… We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds… So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine… so the only other thing that could be causing the problem was pfSense… however I couldn’t find any indication of anything going wrong but the traffic graph… memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc… After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend… and that fixed the problem of requests not getting through and the traffic graph was again stable… I wonder if it is there any known issue with the inbound load balancer… I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~21, as when load testing we tested from a bunch of ~10 IPs… The problem is that we do need load balancing, mainly for redundancy of our systems at the back end… The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we don’t know if pfSense is actually able to take the load… Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONE™ The contents of this e-mail and any attachments/inserts are strictly confidential and sent for the attention of the addressee/s only. This e-mail might contain confidential and/or privileged material therefore if you are not the addressee/s
RE: [pfSense Support] Inbound load balancer performance under heavy load.
From: Jose Hernandez [mailto:j...@vidzone.tv] Sent: June 12, 2009 5:29 AM To: support@pfsense.com Subject: [pfSense Support] Inbound load balancer performance under heavy load. Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time... We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before... We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds... So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine... so the only other thing that could be causing the problem was pfSense... however I couldn't find any indication of anything going wrong but the traffic graph... memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc... After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend... and that fixed the problem of requests not getting through and the traffic graph was again stable... I wonder if it is there any known issue with the inbound load balancer... I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~21, as when load testing we tested from a bunch of ~10 IPs... The problem is that we do need load balancing, mainly for redundancy of our systems at the back end... The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we don't know if pfSense is actually able to take the load... Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONE(tm) What about cpu load at the time the problem was happening? Do you use smp kernel? http://forum.pfsense.org/index.php/topic,6805.0.html Eugene - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
RE: [pfSense Support] Inbound load balancer performance under heavy load.
___ From: Jose Hernandez [mailto:j...@vidzone.tv] Sent: June 12, 2009 5:29 AM To: support@pfsense.com Subject: [pfSense Support] Inbound load balancer performance under heavy load. Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time... We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before... We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds... So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine... so the only other thing that could be causing the problem was pfSense... however I couldn't find any indication of anything going wrong but the traffic graph... memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc... After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend... and that fixed the problem of requests not getting through and the traffic graph was again stable... I wonder if it is there any known issue with the inbound load balancer... I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~21, as when load testing we tested from a bunch of ~10 IPs... The problem is that we do need load balancing, mainly for redundancy of our systems at the back end... The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we don't know if pfSense is actually able to take the load... Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONE(tm) What about cpu load at the time the problem was happening? Do you use smp kernel? http://forum.pfsense.org/index.php/topic,6805.0.html Eugene - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org Hi Eugene, I have attached RRD Graph for the processor load, the problem started at on around 1500 and it was around 1830 when I removed the load balancer. The processor utilization would be higher between 1830 and 2300 because at those time I had the Firewall Optimization Options set up to 'conservative'... I tried that as the problem was packets lost... but as soon as I changed back to 'normal' the processor utilization dropped as expected. Regards, Jose - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
RE: [pfSense Support] Inbound load balancer performance under heavy load.
-Original Message- From: Ermal Luçi [mailto:ermal.l...@gmail.com] Sent: 12 June 2009 12:48 To: support@pfsense.com Subject: Re: [pfSense Support] Inbound load balancer performance under heavy load. Can you please try a later snapshot after 11062009 it seems you had problems with sticky-connections! Though without analysis i might be compeletly WRONG. Ermal On Fri, Jun 12, 2009 at 12:52 PM, Jose Hernandezj...@vidzone.tv wrote: It is fine Tebano, I appreciate your answer and as you said there are not other limitations documented nor any other issues I could find anywhere, and I did spend some time researching Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:44 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 11:33:54 +0100 Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. Thank for your response, however the limitations on the featured list are not the cause of the problem I am happy with the load balancer to equally distribute the load, also happy with the firewall not checking for a valid response but there seems to be any other limitation not listed Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA - Hi Josè. Sorry for the misunderstanding. I was simply trying to say that there aren't other limitation (that I know), except the ones I've pasted You. Probably my answer wasn't really wroten correctly... sorry! Cheers. Tebano. From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:11 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 10:29:03 +0100 Subject: [pfSense Support] Inbound load balancer performance under heavy load. Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine so the only other thing that could be causing the problem was pfSense however I couldnt find any indication of anything going wrong but the traffic graph memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend and that fixed the problem of requests not getting through and the traffic graph was again stable I wonder if it is there any known issue with the inbound load balancer I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~21, as when load testing we tested from a bunch of ~10 IPs The problem is that we do need load balancing, mainly for redundancy of our systems at the back end The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we dont know if pfSense is actually able to take the load Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONE The contents of this e-mail and any attachments/inserts are strictly confidential and sent for the attention of the addressee/s
Re: [pfSense Support] Inbound load balancer performance under heavy load.
On Fri, Jun 12, 2009 at 6:27 PM, Jose Hernandezj...@vidzone.tv wrote: -Original Message- From: Ermal Luçi [mailto:ermal.l...@gmail.com] Sent: 12 June 2009 12:48 To: support@pfsense.com Subject: Re: [pfSense Support] Inbound load balancer performance under heavy load. Can you please try a later snapshot after 11062009 it seems you had problems with sticky-connections! Though without analysis i might be compeletly WRONG. Ermal On Fri, Jun 12, 2009 at 12:52 PM, Jose Hernandezj...@vidzone.tv wrote: It is fine Tebano, I appreciate your answer and as you said there are not other limitations documented nor any other issues I could find anywhere, and I did spend some time researching… Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:44 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 11:33:54 +0100 Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. Thank for your response, however the limitations on the featured list are not the cause of the problem… I am happy with the load balancer to equally distribute the load, also happy with the firewall not checking for a valid response… but there seems to be any other limitation not listed… Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA - Hi Josè. Sorry for the misunderstanding. I was simply trying to say that there aren't other limitation (that I know), except the ones I've pasted You. Probably my answer wasn't really wroten correctly... sorry! Cheers. Tebano. From: Tebano epaminonda [mailto:l_epa_m_ino...@hotmail.com] Sent: 12 June 2009 11:11 To: support@pfsense.com Subject: RE: [pfSense Support] Inbound load balancer performance under heavy load. From: j...@vidzone.tv To: support@pfsense.com Date: Fri, 12 Jun 2009 10:29:03 +0100 Subject: [pfSense Support] Inbound load balancer performance under heavy load. Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time… We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. Yesterday right after we launch the service, we start getting complaints of many requests failing from users. After some investigation it was clear that the request were not getting through to our systems!!! The only indication of something going bad was the traffic graph (attached is a screen grab), it was picking up and down as never before… We did some load testing last week and the week before and we were seeing ~100Mbps constant outbound speed, we also have seen in the past ~100Mbps inbound speeds… So I first blame our IP transit provider, after contacting them, they confirmed to me that no packets were being lost or dropped anywhere in their network and that their systems were just fine… so the only other thing that could be causing the problem was pfSense… however I couldn’t find any indication of anything going wrong but the traffic graph… memory and processor were fine, states table size, no packets dropped in RRD Graphs, etc… After tweaking many settings in pfSense with no joy, I finally removed the Virtual Server and created a NAT Port Forward to only one of our web servers layer at the backend… and that fixed the problem of requests not getting through and the traffic graph was again stable… I wonder if it is there any known issue with the inbound load balancer… I think the problem was with the number of source IPs or states it had to deal with (after the load balancer was removed, the states picked up to ~21, as when load testing we tested from a bunch of ~10 IPs… The problem is that we do need load balancing, mainly for redundancy of our systems at the back end… The inbound load balancer that was set up had 3 servers in the pool and, the port was HTTPS and TCP monitor was configured Is there anything in version 1.2-release that affects the performance of the inbound load balancer? Would this performance issues go away if I upgrade to the latest stable version, currently 1.2.2? We are also thinking in getting commercial support, however we are not sure if this will help as we don’t know if pfSense is actually able to take the load… Can anyone shed some light into this issues we are having? Regards, Jose Hernandez Software and Systems Senior Engineer VIDZONE DIGITAL MEDIA GET IN THE VIDZONE™ The contents of this e-mail
Re: [pfSense Support] Inbound load balancer performance under heavy load.
On Fri, Jun 12, 2009 at 5:29 AM, Jose Hernandezj...@vidzone.tv wrote: Hi, Yesterday we had a service launch, and pfSense inbound load balancer let me down big time… We have been using pfSense 1.2-release version installed on Dell PowerEdge R200 and CARP for redundancy for around a year now, it probed to work although we never have had a very high load. For reasons outside our code base, your FreeBSD 6.2-based version is better for server load balancing than anything based on newer FreeBSD versions. There are regressions we found recently in 7.0 through 7.2, though Ermal may have fixed those, they are not issues in 6.2 to begin with so I would recommend against upgrading especially since Ermal's changes haven't been widely tested yet and this is a production system. It's very hard to say what might be impacting you here, without getting into the system. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
