Re: [squid-users] load balancing and site failover
On Thu, 2015-03-26 at 13:53 +1300, Amos Jeffries wrote: On 26/03/2015 10:26 a.m., Brendan Kearney wrote: On Wed, 2015-03-25 at 15:03 +1300, Amos Jeffries wrote: On 25/03/2015 9:55 a.m., brendan kearney wrote: Was not sure if bugzilla was used for mailing list issues. If you would like me to open one, I will but it looks like the list is working again. Bugzilla is used, list bugs under the project services product. As for your query... On Mar 24, 2015 2:25 PM, Brendan Kearney wrote: On Tue, 2015-03-24 at 10:18 -0400, Brendan Kearney wrote: while load balancing is not a requirement in a proxy environment, it does afford a great deal of functionality, scaling and fault tolerance in one. several if not many on this list probably employ them for their proxies and likely other technologies, but they are not all created equal. i recently looked to see if a specific feature was in HAProxy. i was looking to see if HAProxy could reply to a new connection with a RST packet if no pool member was available. the idea behind this is, if all of the proxies are not passing the service check and are marked down by the load balancer, the reply of a RST in the TCP handshake (i.e. SYN - RST, not SYN - SYN/ACK - ACK) tells the browser to failover to the next proxy assigned by the PAC file. where i work, we have this configuration working. the load balancers are configured with the option to send a reset when no proxy is available in the pool. the PAC file assigns all 4 of the proxy VIPs in a specific order based on which proxy VIP is assigned as the primary. In every case, if the primary VIP does not have an available pool member, the browser fails over to the next in the list. failover would happen again, if the secondary VIP replies with a RST during the connection establishing. the process repeats until a TCP connection establishes or all proxies assigned have been exhausted. the browser will use the proxy VIP that it successfully connects to, for the duration of the session. once the browser is closed and reopened, the evaluation of the PAC file occurs again, and the process starts anew. plug-ins such as Proxy Selector are the exception to this, and can be used to reevaluate a PAC file by selecting it for use. we have used this configuration several times, when we found an ISP link was flapping or some other issue more global in nature than just the proxies was affecting our egress and internet access. i can attest to the solution as working and elegantly handling site wide failures. being that the solutions where i work are proprietary commercial products, i wanted to find an open source product that does this. i have been a long time user of HAProxy, and have recommended it for others here, but sadly they cannot perform this function. per their mailing list, they use the network stack of the OS for connection establishment and cannot cause a RST to be sent to the client during a TCP handshake if no pool member is available. they suggested an external helper that manipulates IPTables rules based on a pool member being available. they do not feel that a feature like this belongs in a layer 4/7 reverse proxy application. They are right. HTTP != TCP. i didnt confuse that detail. it was unknown to me that HAProxy could not tie layer 7 status to layer 3/4 actions. the decisions they made and how they architected the app is why they cannot do this, not that it is technically impossible to do it. i may be spoiled because i work with equipment that can do this for me. In particular TCP depends on routers having a full routing map of the entire Internet (provided by BGP) and deciding the best upstream hop based on that global info. Clients have one (and only one) upstream router for each server they want to connect to. i will contest this. my router does not need a full BGP map to route traffic locally on my LAN or remotely out its WAN interface. hell, it does not even run BGP, and i can still get to the intarwebs, no problem. it too, only has one upstream router / default route. Then your router has more in common with proxy than usual. Its operating with a next-hop packet relay model (OSPF? MPLS?) rather than an end-to-end model (BGP with RIB/FIB). DOCSIS 2 - ethernet on the WAN side and locally connected on the LAN side. :D oh, and a static route pointing a /24 for vpn traffic to a specific device. In HTTP each proxy (aka router) performs independent upstream connection attempts, failover, and verifies it worked before responding to the client with a final response. Each proxy only has enough detail to check its upstream(s). Each proxy can connect to any server (subject to ACLs). how are you comparing a HTTP proxy (a layer 7 application) to a router (a layer 3 device)? routers route traffic and proxies proxy traffic. while, routers
Re: [squid-users] load balancing and site failover
On Wed, 2015-03-25 at 15:03 +1300, Amos Jeffries wrote: On 25/03/2015 9:55 a.m., brendan kearney wrote: Was not sure if bugzilla was used for mailing list issues. If you would like me to open one, I will but it looks like the list is working again. Bugzilla is used, list bugs under the project services product. As for your query... On Mar 24, 2015 2:25 PM, Brendan Kearney wrote: On Tue, 2015-03-24 at 10:18 -0400, Brendan Kearney wrote: while load balancing is not a requirement in a proxy environment, it does afford a great deal of functionality, scaling and fault tolerance in one. several if not many on this list probably employ them for their proxies and likely other technologies, but they are not all created equal. i recently looked to see if a specific feature was in HAProxy. i was looking to see if HAProxy could reply to a new connection with a RST packet if no pool member was available. the idea behind this is, if all of the proxies are not passing the service check and are marked down by the load balancer, the reply of a RST in the TCP handshake (i.e. SYN - RST, not SYN - SYN/ACK - ACK) tells the browser to failover to the next proxy assigned by the PAC file. where i work, we have this configuration working. the load balancers are configured with the option to send a reset when no proxy is available in the pool. the PAC file assigns all 4 of the proxy VIPs in a specific order based on which proxy VIP is assigned as the primary. In every case, if the primary VIP does not have an available pool member, the browser fails over to the next in the list. failover would happen again, if the secondary VIP replies with a RST during the connection establishing. the process repeats until a TCP connection establishes or all proxies assigned have been exhausted. the browser will use the proxy VIP that it successfully connects to, for the duration of the session. once the browser is closed and reopened, the evaluation of the PAC file occurs again, and the process starts anew. plug-ins such as Proxy Selector are the exception to this, and can be used to reevaluate a PAC file by selecting it for use. we have used this configuration several times, when we found an ISP link was flapping or some other issue more global in nature than just the proxies was affecting our egress and internet access. i can attest to the solution as working and elegantly handling site wide failures. being that the solutions where i work are proprietary commercial products, i wanted to find an open source product that does this. i have been a long time user of HAProxy, and have recommended it for others here, but sadly they cannot perform this function. per their mailing list, they use the network stack of the OS for connection establishment and cannot cause a RST to be sent to the client during a TCP handshake if no pool member is available. they suggested an external helper that manipulates IPTables rules based on a pool member being available. they do not feel that a feature like this belongs in a layer 4/7 reverse proxy application. They are right. HTTP != TCP. i didnt confuse that detail. it was unknown to me that HAProxy could not tie layer 7 status to layer 3/4 actions. the decisions they made and how they architected the app is why they cannot do this, not that it is technically impossible to do it. i may be spoiled because i work with equipment that can do this for me. In particular TCP depends on routers having a full routing map of the entire Internet (provided by BGP) and deciding the best upstream hop based on that global info. Clients have one (and only one) upstream router for each server they want to connect to. i will contest this. my router does not need a full BGP map to route traffic locally on my LAN or remotely out its WAN interface. hell, it does not even run BGP, and i can still get to the intarwebs, no problem. it too, only has one upstream router / default route. In HTTP each proxy (aka router) performs independent upstream connection attempts, failover, and verifies it worked before responding to the client with a final response. Each proxy only has enough detail to check its upstream(s). Each proxy can connect to any server (subject to ACLs). how are you comparing a HTTP proxy (a layer 7 application) to a router (a layer 3 device)? routers route traffic and proxies proxy traffic. very different functions. routers dont look past a certain point in the headers in order to make decisions on where to send the traffic. proxies look all the way to the end of the headers and sometimes into the payload, too. proxies are more akin to a protocol specific firewall. proxies also dont send the incoming traffic out an interface. they terminate the client session, and initiate a new session on behalf of the client. simply because the proxy can elect how to send a request it is
Re: [squid-users] load balancing and site failover
On 26/03/2015 10:26 a.m., Brendan Kearney wrote: On Wed, 2015-03-25 at 15:03 +1300, Amos Jeffries wrote: On 25/03/2015 9:55 a.m., brendan kearney wrote: Was not sure if bugzilla was used for mailing list issues. If you would like me to open one, I will but it looks like the list is working again. Bugzilla is used, list bugs under the project services product. As for your query... On Mar 24, 2015 2:25 PM, Brendan Kearney wrote: On Tue, 2015-03-24 at 10:18 -0400, Brendan Kearney wrote: while load balancing is not a requirement in a proxy environment, it does afford a great deal of functionality, scaling and fault tolerance in one. several if not many on this list probably employ them for their proxies and likely other technologies, but they are not all created equal. i recently looked to see if a specific feature was in HAProxy. i was looking to see if HAProxy could reply to a new connection with a RST packet if no pool member was available. the idea behind this is, if all of the proxies are not passing the service check and are marked down by the load balancer, the reply of a RST in the TCP handshake (i.e. SYN - RST, not SYN - SYN/ACK - ACK) tells the browser to failover to the next proxy assigned by the PAC file. where i work, we have this configuration working. the load balancers are configured with the option to send a reset when no proxy is available in the pool. the PAC file assigns all 4 of the proxy VIPs in a specific order based on which proxy VIP is assigned as the primary. In every case, if the primary VIP does not have an available pool member, the browser fails over to the next in the list. failover would happen again, if the secondary VIP replies with a RST during the connection establishing. the process repeats until a TCP connection establishes or all proxies assigned have been exhausted. the browser will use the proxy VIP that it successfully connects to, for the duration of the session. once the browser is closed and reopened, the evaluation of the PAC file occurs again, and the process starts anew. plug-ins such as Proxy Selector are the exception to this, and can be used to reevaluate a PAC file by selecting it for use. we have used this configuration several times, when we found an ISP link was flapping or some other issue more global in nature than just the proxies was affecting our egress and internet access. i can attest to the solution as working and elegantly handling site wide failures. being that the solutions where i work are proprietary commercial products, i wanted to find an open source product that does this. i have been a long time user of HAProxy, and have recommended it for others here, but sadly they cannot perform this function. per their mailing list, they use the network stack of the OS for connection establishment and cannot cause a RST to be sent to the client during a TCP handshake if no pool member is available. they suggested an external helper that manipulates IPTables rules based on a pool member being available. they do not feel that a feature like this belongs in a layer 4/7 reverse proxy application. They are right. HTTP != TCP. i didnt confuse that detail. it was unknown to me that HAProxy could not tie layer 7 status to layer 3/4 actions. the decisions they made and how they architected the app is why they cannot do this, not that it is technically impossible to do it. i may be spoiled because i work with equipment that can do this for me. In particular TCP depends on routers having a full routing map of the entire Internet (provided by BGP) and deciding the best upstream hop based on that global info. Clients have one (and only one) upstream router for each server they want to connect to. i will contest this. my router does not need a full BGP map to route traffic locally on my LAN or remotely out its WAN interface. hell, it does not even run BGP, and i can still get to the intarwebs, no problem. it too, only has one upstream router / default route. Then your router has more in common with proxy than usual. Its operating with a next-hop packet relay model (OSPF? MPLS?) rather than an end-to-end model (BGP with RIB/FIB). In HTTP each proxy (aka router) performs independent upstream connection attempts, failover, and verifies it worked before responding to the client with a final response. Each proxy only has enough detail to check its upstream(s). Each proxy can connect to any server (subject to ACLs). how are you comparing a HTTP proxy (a layer 7 application) to a router (a layer 3 device)? routers route traffic and proxies proxy traffic. while, routers proxy TCP packets and proxies route HTTP messages. Its the behaviour abstraction I'm talking about here. [if you dont want to dive into theory skip to the end of this mail] The algorithms each are capable of are the same despite differences in details of layer and what designed mechanisms are optimal
Re: [squid-users] load balancing and site failover
On 25/03/2015 9:55 a.m., brendan kearney wrote: Was not sure if bugzilla was used for mailing list issues. If you would like me to open one, I will but it looks like the list is working again. Bugzilla is used, list bugs under the project services product. As for your query... On Mar 24, 2015 2:25 PM, Brendan Kearney wrote: On Tue, 2015-03-24 at 10:18 -0400, Brendan Kearney wrote: while load balancing is not a requirement in a proxy environment, it does afford a great deal of functionality, scaling and fault tolerance in one. several if not many on this list probably employ them for their proxies and likely other technologies, but they are not all created equal. i recently looked to see if a specific feature was in HAProxy. i was looking to see if HAProxy could reply to a new connection with a RST packet if no pool member was available. the idea behind this is, if all of the proxies are not passing the service check and are marked down by the load balancer, the reply of a RST in the TCP handshake (i.e. SYN - RST, not SYN - SYN/ACK - ACK) tells the browser to failover to the next proxy assigned by the PAC file. where i work, we have this configuration working. the load balancers are configured with the option to send a reset when no proxy is available in the pool. the PAC file assigns all 4 of the proxy VIPs in a specific order based on which proxy VIP is assigned as the primary. In every case, if the primary VIP does not have an available pool member, the browser fails over to the next in the list. failover would happen again, if the secondary VIP replies with a RST during the connection establishing. the process repeats until a TCP connection establishes or all proxies assigned have been exhausted. the browser will use the proxy VIP that it successfully connects to, for the duration of the session. once the browser is closed and reopened, the evaluation of the PAC file occurs again, and the process starts anew. plug-ins such as Proxy Selector are the exception to this, and can be used to reevaluate a PAC file by selecting it for use. we have used this configuration several times, when we found an ISP link was flapping or some other issue more global in nature than just the proxies was affecting our egress and internet access. i can attest to the solution as working and elegantly handling site wide failures. being that the solutions where i work are proprietary commercial products, i wanted to find an open source product that does this. i have been a long time user of HAProxy, and have recommended it for others here, but sadly they cannot perform this function. per their mailing list, they use the network stack of the OS for connection establishment and cannot cause a RST to be sent to the client during a TCP handshake if no pool member is available. they suggested an external helper that manipulates IPTables rules based on a pool member being available. they do not feel that a feature like this belongs in a layer 4/7 reverse proxy application. They are right. HTTP != TCP. In particular TCP depends on routers having a full routing map of the entire Internet (provided by BGP) and deciding the best upstream hop based on that global info. Clients have one (and only one) upstream router for each server they want to connect to. In HTTP each proxy (aka router) performs independent upstream connection attempts, failover, and verifies it worked before responding to the client with a final response. Each proxy only has enough detail to check its upstream(s). Each proxy can connect to any server (subject to ACLs). my search for a load balancer solution went through ipvsadm, balance and haproxy before i selected haproxy. haproxy was more feature rich than balance, and easier to implement than ipvsadm. do any other list members have a need for such a feature from their load balancers? do any other list members have site failover solutions that have been tested or used and would consider sharing their design and/or pain points? i am not looking for secret sauce or confidential info, but more high level architecture decisions and such. I havent tested it but this should do what you are asking: acl err http_status 500-505 408 deny_info TCP_RESET err http_reply_access deny err It replaces the response from Squid with a TCP RST packet. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] load balancing and site failover
Was not sure if bugzilla was used for mailing list issues. If you would like me to open one, I will but it looks like the list is working again. On Mar 24, 2015 2:25 PM, Brendan Kearney bpk...@gmail.com wrote: On Tue, 2015-03-24 at 10:18 -0400, Brendan Kearney wrote: while load balancing is not a requirement in a proxy environment, it does afford a great deal of functionality, scaling and fault tolerance in one. several if not many on this list probably employ them for their proxies and likely other technologies, but they are not all created equal. i recently looked to see if a specific feature was in HAProxy. i was looking to see if HAProxy could reply to a new connection with a RST packet if no pool member was available. the idea behind this is, if all of the proxies are not passing the service check and are marked down by the load balancer, the reply of a RST in the TCP handshake (i.e. SYN - RST, not SYN - SYN/ACK - ACK) tells the browser to failover to the next proxy assigned by the PAC file. where i work, we have this configuration working. the load balancers are configured with the option to send a reset when no proxy is available in the pool. the PAC file assigns all 4 of the proxy VIPs in a specific order based on which proxy VIP is assigned as the primary. In every case, if the primary VIP does not have an available pool member, the browser fails over to the next in the list. failover would happen again, if the secondary VIP replies with a RST during the connection establishing. the process repeats until a TCP connection establishes or all proxies assigned have been exhausted. the browser will use the proxy VIP that it successfully connects to, for the duration of the session. once the browser is closed and reopened, the evaluation of the PAC file occurs again, and the process starts anew. plug-ins such as Proxy Selector are the exception to this, and can be used to reevaluate a PAC file by selecting it for use. we have used this configuration several times, when we found an ISP link was flapping or some other issue more global in nature than just the proxies was affecting our egress and internet access. i can attest to the solution as working and elegantly handling site wide failures. being that the solutions where i work are proprietary commercial products, i wanted to find an open source product that does this. i have been a long time user of HAProxy, and have recommended it for others here, but sadly they cannot perform this function. per their mailing list, they use the network stack of the OS for connection establishment and cannot cause a RST to be sent to the client during a TCP handshake if no pool member is available. they suggested an external helper that manipulates IPTables rules based on a pool member being available. they do not feel that a feature like this belongs in a layer 4/7 reverse proxy application. my search for a load balancer solution went through ipvsadm, balance and haproxy before i selected haproxy. haproxy was more feature rich than balance, and easier to implement than ipvsadm. do any other list members have a need for such a feature from their load balancers? do any other list members have site failover solutions that have been tested or used and would consider sharing their design and/or pain points? i am not looking for secret sauce or confidential info, but more high level architecture decisions and such. trying to send this again, as it was rejected previously. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
