Logan,
My 2cents,
While many including myself dont qualify as developers on this list,
there are some development parts anyone can do - without knowing Java.
Specifically, agentless config management does not require much or any
java programing in first stage. Perhaps the portion where a config file
for vRouter needs to be fetched and apply by dfault - needs some
enhancement, everything else can be handled by savy system person.
With that said, it should be trivial to add agent-less config management
to router vm without any hardcore java expertise.
Regards
ilya
On 4/1/15 10:13 AM, Logan Barfield wrote:
Hi Ilya,
I definitely like that solution, and there is discussion elsewhere on
overhauling how VRs are configured.
Unfortunately I am not a developer. Any changes I make are generally
to existing code, and are based off of context and testing more than
anything else. I don't currently have the time or knowledge required
to implement such a big change myself, and even if I did I wouldn't
know where to start. I would be more than happy to discuss something
like that with other community members, and offer patches/testing
where I can. I don't see myself being able to take the lead on such a
task though.
For now I'd like to address this particular issue (transparent proxy
support), to see if the community has any objections or concerns about
the proposed changes. We can (and probably will) end up just creating
our own internal branch, but I'd rather be able to commit any changes
we make to the project for ease of upgrades, support, etc.
Thank You,
Logan Barfield
Tranquil Hosting
On Wed, Apr 1, 2015 at 12:58 PM, ilya <ilya.mailing.li...@gmail.com> wrote:
Logan
For sometime now, idea was kicked around of extending VR services where it
can take custom configurations.
In theory, VR should be driven via config management tool (i.e. puppet,
salt, chef, ansible - or whatever works).
Instead of hacking this in portions, perhaps you can spend sometime
analizing how VR is build and how you we can extend the functionality via
external config management.
For example, when vrouter comes up, because we already have ssh keys dropped
on each VR, just on this premise alone - we can do just about anything. I'm
thinking salt or ansible would be a better solution as it is SSH driven and
agentless.
Try porting your changes into salt or ansible (or anything else agentless)
and propose a framework on how we can enhance this.
I'm thinking, for each Account (as it gets a VR), we need to have a ansible
config file that will be pushed through once VR comes up to configure
remaining custom bits/components to your liking.
Thoughts?
regards
ilya
On 4/1/15 9:34 AM, Logan Barfield wrote:
We've been running into some issues with the Advanced Zone/Isolated
Network Load Balancer, and in working through them we've come up with
some ideas for how the functionality can be improved.
The first issue we hit was with HTTP load balancing. We had a site
that was sending back larger than average HTTP headers with 302
redirects. This was causing HAProxy to return 502 errors to clients.
This is apparently a known issue with HAProxy when using the default
"tune.bufsize" and "tune.maxrewrite" settings.The official HAProxy
documentation recommends changing these from the defaults. We were
able to work around the problem by manually setting "tune.maxrewrite
1024" in the haproxy.cfg on the virtual router. This resolved most of
the 502 errors, and would have probably resolved all of them with more
tuning. The problem is that this change obviously wouldn't survive
upgrades or VR rebuilds.
To fix the problem on a more permanent basis we changed the
KeepAliveEnabled Network Offering setting introduced a few versions
ago. This directs HAProxy to use TCP mode instead of HTTP mode for
rules configured on port 80. This solution works for the most part,
but there are a couple of problems:
1) There doesn't appear to be support for this setting in the UI.
That's understandable as the UI is way behind the current feature set.
2) There doesn't appear to be support for this setting in the API,
either when creating or updating Network Offerings. This is a bit of
a problem. We had to make the change in the database directly, which
is very dirty.
3) TCP connections don't grab HTTP headers, so they can't send the
real IP to HTTP/Nginx in the X-Forwarded-For header.
To fix these issues I suggest the following changes be made:
1) Add the "KeepAliveEnabled" option to the Network Offering API
commands. I really have no idea how to do this, so I'll try to flag
the original committer to see if they can do so.
2) Add a new option for TPROXY support. The current VR kernel and
HAProxy version have TPROXY support built in, so having the option (on
a per LB rule basis) would be great. This would allow for using TCP
mode in HAProxy, while still passing the real IP through to the
backend services. To accomplish this I would suggest adding the
necessary IPtables rules to the VR either by default, or when Load
Balancing is first enabled. Then a flag can be added to the create LB
rule command to either enable or disable the transparent proxy
setting.
The necessary IPtables rules are:
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
Then for each LB rule created with the "transparent" option enabled,
add the following setting to the rule configuration:
source 0.0.0.0 usesrc clientip
The last configuration change is to remove the user/group or uid/gid
options in the haproxy config, otherwise HAProxy won't start with
"usesrc" enabled.
It is also recommended to enable
/proc/sys/net/ipv4/conf/eth0/send_redirects, but I haven't noticed any
issues with it disabled either, and I don't know what else it might
affect.
3) The last suggestion would be to move away from hard coding
configuration directives for VR services (like HAProxy:
https://github.com/apache/cloudstack/blob/5091d0f5c5b03cb8658f2d974103261341080825/core/src/com/cloud/network/HAProxyConfigurator.java)
Doing this makes implementing changes a hassle since it involves
rebuilding/upgrading CloudStack to accomplish anything, even small
edits. For a production environment this is ill advised if not
impossible. In general it would make sense to make persistent changes
to VR services possible without recompiling code or rebooting the VRs.
I believe that's part of a bigger issue though, as I've seen some
discussion about it on the list.
If anyone actually makes it through this, I'd appreciate any feedback
on things I may not be considering, or reasons not to implement these
changes. I doubt I'll get enough traction for an actual developer to
help, so I'll probably end up hacking these in myself and committing
them. I just wanted to see what the community thought first.
Thank You,
Logan Barfield
Tranquil Hosting