Load Balancer (HAProxy) - TPROXY passthrough
We've been running into some issues with the Advanced Zone/Isolated Network Load Balancer, and in working through them we've come up with some ideas for how the functionality can be improved. The first issue we hit was with HTTP load balancing. We had a site that was sending back larger than average HTTP headers with 302 redirects. This was causing HAProxy to return 502 errors to clients. This is apparently a known issue with HAProxy when using the default "tune.bufsize" and "tune.maxrewrite" settings.The official HAProxy documentation recommends changing these from the defaults. We were able to work around the problem by manually setting "tune.maxrewrite 1024" in the haproxy.cfg on the virtual router. This resolved most of the 502 errors, and would have probably resolved all of them with more tuning. The problem is that this change obviously wouldn't survive upgrades or VR rebuilds. To fix the problem on a more permanent basis we changed the KeepAliveEnabled Network Offering setting introduced a few versions ago. This directs HAProxy to use TCP mode instead of HTTP mode for rules configured on port 80. This solution works for the most part, but there are a couple of problems: 1) There doesn't appear to be support for this setting in the UI. That's understandable as the UI is way behind the current feature set. 2) There doesn't appear to be support for this setting in the API, either when creating or updating Network Offerings. This is a bit of a problem. We had to make the change in the database directly, which is very dirty. 3) TCP connections don't grab HTTP headers, so they can't send the real IP to HTTP/Nginx in the X-Forwarded-For header. To fix these issues I suggest the following changes be made: 1) Add the "KeepAliveEnabled" option to the Network Offering API commands. I really have no idea how to do this, so I'll try to flag the original committer to see if they can do so. 2) Add a new option for TPROXY support. The current VR kernel and HAProxy version have TPROXY support built in, so having the option (on a per LB rule basis) would be great. This would allow for using TCP mode in HAProxy, while still passing the real IP through to the backend services. To accomplish this I would suggest adding the necessary IPtables rules to the VR either by default, or when Load Balancing is first enabled. Then a flag can be added to the create LB rule command to either enable or disable the transparent proxy setting. The necessary IPtables rules are: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 111 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 111 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 Then for each LB rule created with the "transparent" option enabled, add the following setting to the rule configuration: source 0.0.0.0 usesrc clientip The last configuration change is to remove the user/group or uid/gid options in the haproxy config, otherwise HAProxy won't start with "usesrc" enabled. It is also recommended to enable /proc/sys/net/ipv4/conf/eth0/send_redirects, but I haven't noticed any issues with it disabled either, and I don't know what else it might affect. 3) The last suggestion would be to move away from hard coding configuration directives for VR services (like HAProxy: https://github.com/apache/cloudstack/blob/5091d0f5c5b03cb8658f2d974103261341080825/core/src/com/cloud/network/HAProxyConfigurator.java) Doing this makes implementing changes a hassle since it involves rebuilding/upgrading CloudStack to accomplish anything, even small edits. For a production environment this is ill advised if not impossible. In general it would make sense to make persistent changes to VR services possible without recompiling code or rebooting the VRs. I believe that's part of a bigger issue though, as I've seen some discussion about it on the list. If anyone actually makes it through this, I'd appreciate any feedback on things I may not be considering, or reasons not to implement these changes. I doubt I'll get enough traction for an actual developer to help, so I'll probably end up hacking these in myself and committing them. I just wanted to see what the community thought first. Thank You, Logan Barfield Tranquil Hosting
Re: Load Balancer (HAProxy) - TPROXY passthrough
Logan For sometime now, idea was kicked around of extending VR services where it can take custom configurations. In theory, VR should be driven via config management tool (i.e. puppet, salt, chef, ansible - or whatever works). Instead of hacking this in portions, perhaps you can spend sometime analizing how VR is build and how you we can extend the functionality via external config management. For example, when vrouter comes up, because we already have ssh keys dropped on each VR, just on this premise alone - we can do just about anything. I'm thinking salt or ansible would be a better solution as it is SSH driven and agentless. Try porting your changes into salt or ansible (or anything else agentless) and propose a framework on how we can enhance this. I'm thinking, for each Account (as it gets a VR), we need to have a ansible config file that will be pushed through once VR comes up to configure remaining custom bits/components to your liking. Thoughts? regards ilya On 4/1/15 9:34 AM, Logan Barfield wrote: We've been running into some issues with the Advanced Zone/Isolated Network Load Balancer, and in working through them we've come up with some ideas for how the functionality can be improved. The first issue we hit was with HTTP load balancing. We had a site that was sending back larger than average HTTP headers with 302 redirects. This was causing HAProxy to return 502 errors to clients. This is apparently a known issue with HAProxy when using the default "tune.bufsize" and "tune.maxrewrite" settings.The official HAProxy documentation recommends changing these from the defaults. We were able to work around the problem by manually setting "tune.maxrewrite 1024" in the haproxy.cfg on the virtual router. This resolved most of the 502 errors, and would have probably resolved all of them with more tuning. The problem is that this change obviously wouldn't survive upgrades or VR rebuilds. To fix the problem on a more permanent basis we changed the KeepAliveEnabled Network Offering setting introduced a few versions ago. This directs HAProxy to use TCP mode instead of HTTP mode for rules configured on port 80. This solution works for the most part, but there are a couple of problems: 1) There doesn't appear to be support for this setting in the UI. That's understandable as the UI is way behind the current feature set. 2) There doesn't appear to be support for this setting in the API, either when creating or updating Network Offerings. This is a bit of a problem. We had to make the change in the database directly, which is very dirty. 3) TCP connections don't grab HTTP headers, so they can't send the real IP to HTTP/Nginx in the X-Forwarded-For header. To fix these issues I suggest the following changes be made: 1) Add the "KeepAliveEnabled" option to the Network Offering API commands. I really have no idea how to do this, so I'll try to flag the original committer to see if they can do so. 2) Add a new option for TPROXY support. The current VR kernel and HAProxy version have TPROXY support built in, so having the option (on a per LB rule basis) would be great. This would allow for using TCP mode in HAProxy, while still passing the real IP through to the backend services. To accomplish this I would suggest adding the necessary IPtables rules to the VR either by default, or when Load Balancing is first enabled. Then a flag can be added to the create LB rule command to either enable or disable the transparent proxy setting. The necessary IPtables rules are: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 111 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 111 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 Then for each LB rule created with the "transparent" option enabled, add the following setting to the rule configuration: source 0.0.0.0 usesrc clientip The last configuration change is to remove the user/group or uid/gid options in the haproxy config, otherwise HAProxy won't start with "usesrc" enabled. It is also recommended to enable /proc/sys/net/ipv4/conf/eth0/send_redirects, but I haven't noticed any issues with it disabled either, and I don't know what else it might affect. 3) The last suggestion would be to move away from hard coding configuration directives for VR services (like HAProxy: https://github.com/apache/cloudstack/blob/5091d0f5c5b03cb8658f2d974103261341080825/core/src/com/cloud/network/HAProxyConfigurator.java) Doing this makes implementing changes a hassle since it involves rebuilding/upgrading CloudStack to accomplish anything, even small edits. For a production environment this is ill advised if not impossible. In general it would make sense to make persistent changes to VR services possible without recompiling code or rebooting the VRs. I believe that's part of a bigger issue though, as I've seen some discussion about it on the
Re: Load Balancer (HAProxy) - TPROXY passthrough
Hi Ilya, I definitely like that solution, and there is discussion elsewhere on overhauling how VRs are configured. Unfortunately I am not a developer. Any changes I make are generally to existing code, and are based off of context and testing more than anything else. I don't currently have the time or knowledge required to implement such a big change myself, and even if I did I wouldn't know where to start. I would be more than happy to discuss something like that with other community members, and offer patches/testing where I can. I don't see myself being able to take the lead on such a task though. For now I'd like to address this particular issue (transparent proxy support), to see if the community has any objections or concerns about the proposed changes. We can (and probably will) end up just creating our own internal branch, but I'd rather be able to commit any changes we make to the project for ease of upgrades, support, etc. Thank You, Logan Barfield Tranquil Hosting On Wed, Apr 1, 2015 at 12:58 PM, ilya wrote: > Logan > > For sometime now, idea was kicked around of extending VR services where it > can take custom configurations. > > In theory, VR should be driven via config management tool (i.e. puppet, > salt, chef, ansible - or whatever works). > > Instead of hacking this in portions, perhaps you can spend sometime > analizing how VR is build and how you we can extend the functionality via > external config management. > > For example, when vrouter comes up, because we already have ssh keys dropped > on each VR, just on this premise alone - we can do just about anything. I'm > thinking salt or ansible would be a better solution as it is SSH driven and > agentless. > > Try porting your changes into salt or ansible (or anything else agentless) > and propose a framework on how we can enhance this. > > I'm thinking, for each Account (as it gets a VR), we need to have a ansible > config file that will be pushed through once VR comes up to configure > remaining custom bits/components to your liking. > > Thoughts? > > regards > ilya > > > On 4/1/15 9:34 AM, Logan Barfield wrote: >> >> We've been running into some issues with the Advanced Zone/Isolated >> Network Load Balancer, and in working through them we've come up with >> some ideas for how the functionality can be improved. >> >> The first issue we hit was with HTTP load balancing. We had a site >> that was sending back larger than average HTTP headers with 302 >> redirects. This was causing HAProxy to return 502 errors to clients. >> >> This is apparently a known issue with HAProxy when using the default >> "tune.bufsize" and "tune.maxrewrite" settings.The official HAProxy >> documentation recommends changing these from the defaults. We were >> able to work around the problem by manually setting "tune.maxrewrite >> 1024" in the haproxy.cfg on the virtual router. This resolved most of >> the 502 errors, and would have probably resolved all of them with more >> tuning. The problem is that this change obviously wouldn't survive >> upgrades or VR rebuilds. >> >> To fix the problem on a more permanent basis we changed the >> KeepAliveEnabled Network Offering setting introduced a few versions >> ago. This directs HAProxy to use TCP mode instead of HTTP mode for >> rules configured on port 80. This solution works for the most part, >> but there are a couple of problems: >> 1) There doesn't appear to be support for this setting in the UI. >> That's understandable as the UI is way behind the current feature set. >> 2) There doesn't appear to be support for this setting in the API, >> either when creating or updating Network Offerings. This is a bit of >> a problem. We had to make the change in the database directly, which >> is very dirty. >> 3) TCP connections don't grab HTTP headers, so they can't send the >> real IP to HTTP/Nginx in the X-Forwarded-For header. >> >> To fix these issues I suggest the following changes be made: >> 1) Add the "KeepAliveEnabled" option to the Network Offering API >> commands. I really have no idea how to do this, so I'll try to flag >> the original committer to see if they can do so. >> >> 2) Add a new option for TPROXY support. The current VR kernel and >> HAProxy version have TPROXY support built in, so having the option (on >> a per LB rule basis) would be great. This would allow for using TCP >> mode in HAProxy, while still passing the real IP through to the >> backend services. To accomplish this I would suggest adding the >> necessary IPtables rules to the VR either by default, or when Load >> Balancing is first enabled. Then a flag can be added to the create LB >> rule command to either enable or disable the transparent proxy >> setting. >> The necessary IPtables rules are: >> >> iptables -t mangle -N DIVERT >> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT >> iptables -t mangle -A DIVERT -j MARK --set-mark 111 >> iptables -t mangle -A DIVERT -j ACCEPT >> ip rule add fwmark 11
Re: Load Balancer (HAProxy) - TPROXY passthrough
Logan, My 2cents, While many including myself dont qualify as developers on this list, there are some development parts anyone can do - without knowing Java. Specifically, agentless config management does not require much or any java programing in first stage. Perhaps the portion where a config file for vRouter needs to be fetched and apply by dfault - needs some enhancement, everything else can be handled by savy system person. With that said, it should be trivial to add agent-less config management to router vm without any hardcore java expertise. Regards ilya On 4/1/15 10:13 AM, Logan Barfield wrote: Hi Ilya, I definitely like that solution, and there is discussion elsewhere on overhauling how VRs are configured. Unfortunately I am not a developer. Any changes I make are generally to existing code, and are based off of context and testing more than anything else. I don't currently have the time or knowledge required to implement such a big change myself, and even if I did I wouldn't know where to start. I would be more than happy to discuss something like that with other community members, and offer patches/testing where I can. I don't see myself being able to take the lead on such a task though. For now I'd like to address this particular issue (transparent proxy support), to see if the community has any objections or concerns about the proposed changes. We can (and probably will) end up just creating our own internal branch, but I'd rather be able to commit any changes we make to the project for ease of upgrades, support, etc. Thank You, Logan Barfield Tranquil Hosting On Wed, Apr 1, 2015 at 12:58 PM, ilya wrote: Logan For sometime now, idea was kicked around of extending VR services where it can take custom configurations. In theory, VR should be driven via config management tool (i.e. puppet, salt, chef, ansible - or whatever works). Instead of hacking this in portions, perhaps you can spend sometime analizing how VR is build and how you we can extend the functionality via external config management. For example, when vrouter comes up, because we already have ssh keys dropped on each VR, just on this premise alone - we can do just about anything. I'm thinking salt or ansible would be a better solution as it is SSH driven and agentless. Try porting your changes into salt or ansible (or anything else agentless) and propose a framework on how we can enhance this. I'm thinking, for each Account (as it gets a VR), we need to have a ansible config file that will be pushed through once VR comes up to configure remaining custom bits/components to your liking. Thoughts? regards ilya On 4/1/15 9:34 AM, Logan Barfield wrote: We've been running into some issues with the Advanced Zone/Isolated Network Load Balancer, and in working through them we've come up with some ideas for how the functionality can be improved. The first issue we hit was with HTTP load balancing. We had a site that was sending back larger than average HTTP headers with 302 redirects. This was causing HAProxy to return 502 errors to clients. This is apparently a known issue with HAProxy when using the default "tune.bufsize" and "tune.maxrewrite" settings.The official HAProxy documentation recommends changing these from the defaults. We were able to work around the problem by manually setting "tune.maxrewrite 1024" in the haproxy.cfg on the virtual router. This resolved most of the 502 errors, and would have probably resolved all of them with more tuning. The problem is that this change obviously wouldn't survive upgrades or VR rebuilds. To fix the problem on a more permanent basis we changed the KeepAliveEnabled Network Offering setting introduced a few versions ago. This directs HAProxy to use TCP mode instead of HTTP mode for rules configured on port 80. This solution works for the most part, but there are a couple of problems: 1) There doesn't appear to be support for this setting in the UI. That's understandable as the UI is way behind the current feature set. 2) There doesn't appear to be support for this setting in the API, either when creating or updating Network Offerings. This is a bit of a problem. We had to make the change in the database directly, which is very dirty. 3) TCP connections don't grab HTTP headers, so they can't send the real IP to HTTP/Nginx in the X-Forwarded-For header. To fix these issues I suggest the following changes be made: 1) Add the "KeepAliveEnabled" option to the Network Offering API commands. I really have no idea how to do this, so I'll try to flag the original committer to see if they can do so. 2) Add a new option for TPROXY support. The current VR kernel and HAProxy version have TPROXY support built in, so having the option (on a per LB rule basis) would be great. This would allow for using TCP mode in HAProxy, while still passing the real IP through to the backend services. To accomplish this I would suggest adding the necessary IPtables rul
