lol.... good find.. -- Sent from the Delta quadrant using Borg technology!
Nux! www.nux.ro ----- Original Message ----- > From: "Pierre-Luc Dion" <pd...@cloudops.com> > To: "Wido den Hollander" <w...@widodh.nl> > Cc: "dev" <d...@cloudstack.apache.org>, "users" <users@cloudstack.apache.org> > Sent: Friday, 17 November, 2017 13:56:11 > Subject: Re: HTTPS LB and x-forwarded-for > That's funny! > > Cloudstack ui does not provide lb protocol options, but the api does and > cloudstack already support proxy proto v1!!! > > So that's cool! > > Le 13 nov. 2017 09 h 18, "Wido den Hollander" <w...@widodh.nl> a écrit : > >> >> > Op 13 november 2017 om 15:14 schreef Pierre-Luc Dion <pd...@cloudops.com >> >: >> > >> > >> > Hi, >> > >> > This is looking quite promising, I have a colleague that tested that last >> > Friday, look like the proxy proto v1 work out of the box with Nginx, but >> > would need an extra package for Apache 2.4 ? >> >> It depends. You need HTTPd 2.4.28, see: https://httpd.apache.org/docs/ >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol >> >> It's there upstream, but not in all packages. >> >> It can from this module: >> >> - https://github.com/roadrunner2/mod-proxy-protocol >> - https://roadrunner2.github.io/mod-proxy-protocol/mod_proxy_protocol.html >> >> They donated the code to go upstream and went into mod_remoteip but landed >> in 2.4.28 >> >> It will probably make it into Ubuntu 18.04 and CentOS 7.4. >> >> Wido >> >> > Otherwise, it seems to be a good way to do https LB without complicated >> > configuration and huge changes in CloudStack. >> > >> > >> > >> > On Fri, Nov 10, 2017 at 10:36 AM, Nux! <n...@li.nux.ro> wrote: >> > >> > > Pierre-Luc, >> > > >> > > Haproxy docs say it should work for any kind of traffic as long as both >> > > ends are PROXY-aware and it look like a majority of software is. >> > > So, in short, yes. >> > > >> > > -- >> > > Sent from the Delta quadrant using Borg technology! >> > > >> > > Nux! >> > > www.nux.ro >> > > >> > > ----- Original Message ----- >> > > > From: "Pierre-Luc Dion" <pd...@cloudops.com> >> > > > To: "Wido den Hollander" <w...@widodh.nl> >> > > > Cc: "dev" <d...@cloudstack.apache.org>, "Khosrow Moossavi" < >> > > kmooss...@cloudops.com>, "Will Stevens" >> > > > <wstev...@cloudops.com>, "Nux!" <n...@li.nux.ro>, "users" < >> > > users@cloudstack.apache.org> >> > > > Sent: Friday, 10 November, 2017 15:32:38 >> > > > Subject: Re: HTTPS LB and x-forwarded-for >> > > >> > > > Hi Wido, do you know if this would work for https traffic too? >> > > > >> > > > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <w...@widodh.nl> a >> écrit : >> > > > >> > > >> >> > > >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion < >> > > pd...@cloudops.com >> > > >> >: >> > > >> > >> > > >> > >> > > >> > I kind of like the proxy backend type, ill check on our end if >> that >> > > would >> > > >> > work but definitely a simple and efficient approach! >> > > >> > >> > > >> >> > > >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/ >> > > >> >> > > >> Apache HTTPd supports PROXY since 2.4.28: >> > > https://httpd.apache.org/docs/ >> > > >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol >> > > >> >> > > >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer" >> > > >> >> > > >> Wido >> > > >> >> > > >> > >> > > >> > >> > > >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <w...@widodh.nl> a >> > > écrit : >> > > >> > >> > > >> > > >> > > >> > > > Op 9 november 2017 om 19:59 schreef Nux! <n...@li.nux.ro>: >> > > >> > > > >> > > >> > > > >> > > >> > > > Wido, >> > > >> > > > >> > > >> > > > Excellent suggestion with the "transparent proxy", I was not >> > > aware of >> > > >> > > that. >> > > >> > > > I think that would be a great idea and wouldn't require too >> many >> > > >> > > modifications, especially as Haproxy comes already with the VR. >> > > >> > > > >> > > >> > > >> > > >> > > It's indeed just a matter of a HAProxy config setting. We could >> > > make it >> > > >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY >> for >> > > >> example. >> > > >> > > >> > > >> > > That way your problem would be solved. >> > > >> > > >> > > >> > > Wido >> > > >> > > >> > > >> > > > To Paul: >> > > >> > > > - imho the LB solution ACS ships now is a bit handicaped since >> > > you do >> > > >> > > not know the remote host ip. You're flying blind unless you use >> > > google >> > > >> > > analytics (and these things have gotten more and more >> aggressively >> > > >> filtered >> > > >> > > by adblocks). >> > > >> > > > Enhancing Haproxy as Wido suggested would go a long way, it >> > > wouldn't >> > > >> > > break existing functionality and would also keep SSL processing >> off >> > > >> the VR. >> > > >> > > > >> > > >> > > > -- >> > > >> > > > Sent from the Delta quadrant using Borg technology! >> > > >> > > > >> > > >> > > > Nux! >> > > >> > > > www.nux.ro >> > > >> > > > >> > > >> > > > ----- Original Message ----- >> > > >> > > > > From: "Andrija Panic" <andrija.pa...@gmail.com> >> > > >> > > > > To: "users" <users@cloudstack.apache.org> >> > > >> > > > > Cc: "Khosrow Moossavi" <kmooss...@cloudops.com>, "Will >> > > Stevens" < >> > > >> > > wstev...@cloudops.com>, "dev" >> > > >> > > > > <d...@cloudstack.apache.org>, "Pierre-Luc Dion" < >> > > pd...@cloudops.com >> > > >> > >> > > >> > > > > Sent: Thursday, 9 November, 2017 13:10:58 >> > > >> > > > > Subject: Re: HTTPS LB and x-forwarded-for >> > > >> > > > >> > > >> > > > > Wido, >> > > >> > > > > >> > > >> > > > > backend servers are not Linux only, for example we have a >> ton of >> > > >> > > Windows >> > > >> > > > > customers, some WEB solutions / IIS etc... >> > > >> > > > > >> > > >> > > > > @all - If we try to please/solve everyone's proxying >> > > >> > > solution/requirement - >> > > >> > > > > this is impossible IMHO - I'm thinking more about some "do >> it as >> > > >> you >> > > >> > > like" >> > > >> > > > > solution, to let customer write his own haproxy config and >> > > upoad it >> > > >> > > (for >> > > >> > > > > example, or something better?). >> > > >> > > > > >> > > >> > > > > We can support newer version of haproxy (1.5+) which also >> > > implement >> > > >> > > > > "transarent proxy" (integrate with kernel so to speak) to >> allow >> > > >> > > TCP-level >> > > >> > > > > connections to backend (TCP mode, not HTTP mode) but to >> still >> > > >> > > "preserve" >> > > >> > > > > remote IP by faking it (fake soruce IP = transarent proxy). >> > > >> > > > > >> > > >> > > > > For the rest of configuration options, I would leave it to >> the >> > > >> > > customer >> > > >> > > > > how he/she wants to configure rest of haproxy configuration, >> > > >> inlcuding >> > > >> > > > > custom checks, etc. Haproxy configuration is never-ending >> story, >> > > >> and we >> > > >> > > > > probably should allow custom sripts/configuration instead of >> > > >> trying to >> > > >> > > > > provide GUI/API way to configure everything (which is >> > > >> impossible...) >> > > >> > > > > >> > > >> > > > > Just my 2 cents... >> > > >> > > > > >> > > >> > > > > On 9 November 2017 at 08:13, Wido den Hollander < >> w...@widodh.nl >> > > > >> > > >> > > wrote: >> > > >> > > > > >> > > >> > > > >> >> > > >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion < >> > > >> > > pd...@cloudops.com >> > > >> > > > >> >: >> > > >> > > > >> > >> > > >> > > > >> > >> > > >> > > > >> > Same challenge here too! >> > > >> > > > >> > >> > > >> > > > >> > Let's look at improving Load-balancing offering from >> > > >> cloudstack, I >> > > >> > > guest >> > > >> > > > >> we >> > > >> > > > >> > should do a feature spec draft soon.., from my >> perspective, >> > > >> doing >> > > >> > > SSL >> > > >> > > > >> > offload on the VR could be problematic if the VR spec if >> too >> > > >> small, >> > > >> > > and >> > > >> > > > >> the >> > > >> > > > >> > default spec of the VR being 1vcpu@256MB, considering >> it can >> > > >> be the >> > > >> > > > >> router >> > > >> > > > >> > of a VPC, doing VPN termination, adding HTTPS is a bit >> > > ish... >> > > >> What >> > > >> > > would >> > > >> > > > >> > be your thought about this ? >> > > >> > > > >> > >> > > >> > > > >> > I'd be curious to have a LB offering in ACS where it >> would >> > > >> deploy a >> > > >> > > > >> > redundant traefik[1] beside the VR for doing http and >> https >> > > >> > > > >> Load-balancing. >> > > >> > > > >> > I think it would also be useful if the API of that >> traefik >> > > >> instance >> > > >> > > would >> > > >> > > > >> > be available from within the VPC or LBnetwork so is API >> > > would be >> > > >> > > > >> accessible >> > > >> > > > >> > to other apps orchestration tools such as kubernetes or >> > > >> rancher. >> > > >> > > > >> > >> > > >> > > > >> > traefik or not, here is what I think is needed by >> cloudstack >> > > in >> > > >> the >> > > >> > > LB >> > > >> > > > >> > improvement: >> > > >> > > > >> > >> > > >> > > > >> > - support http, https (X-Forwarded-For) >> > > >> > > > >> >> > > >> > > > >> HAProxy also supports the PROXY protocol towards the >> backends. >> > > >> Apache >> > > >> > > > >> 2.4.22 supports this natively and Varnish for example can >> also >> > > >> talk >> > > >> > > PROXY. >> > > >> > > > >> >> > > >> > > > >> It adds a littlebit of metadata to the connection so that >> the >> > > >> backend >> > > >> > > > >> knows the original IP the connection came from for example: >> > > >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy- >> protocol.txt >> > > >> > > > >> >> > > >> > > > >> Wido >> > > >> > > > >> >> > > >> > > > >> > - basic persistence tuning (API already exist) >> > > >> > > > >> > - better backend monitoring, currently only a tcp connect >> > > >> validate >> > > >> > > if the >> > > >> > > > >> > webserver is up. >> > > >> > > > >> > - ssl offload >> > > >> > > > >> > - metric collection, more stats, maybe just export the >> tool >> > > >> status >> > > >> > > page >> > > >> > > > >> to >> > > >> > > > >> > the private network. >> > > >> > > > >> > - Container world support, right now if you have Rancher >> or >> > > >> > > kubernetes >> > > >> > > > >> > cluster, you need to deploy your own LB solution behing >> > > >> mostlikely a >> > > >> > > > >> static >> > > >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub >> or >> > > >> Rancher >> > > >> > > could >> > > >> > > > >> > reuse this instance and managed it to properly do LB >> between >> > > >> > > containers. >> > > >> > > > >> > >> > > >> > > > >> > >> > > >> > > > >> > What would be your prefered LB tool: >> > > >> > > > >> > haproxy, traefik or nginx? >> > > >> > > > >> > >> > > >> > > > >> > CloudStack already have to code to handle SSL certs per >> > > >> projects and >> > > >> > > > >> > accounts if not mistaking because that code was added to >> > > support >> > > >> > > > >> NetScaler >> > > >> > > > >> > as Load-balancer in the past. so one less thing to think >> > > about >> > > >> :-) >> > > >> > > > >> > >> > > >> > > > >> > >> > > >> > > > >> > [1] https://traefik.io/ >> > > >> > > > >> > >> > > >> > > > >> > >> > > >> > > > >> > PL, >> > > >> > > > >> > >> > > >> > > > >> > >> > > >> > > > >> > >> > > >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <n...@li.nux.ro> >> wrote: >> > > >> > > > >> > >> > > >> > > > >> > > Thanks Andrija, >> > > >> > > > >> > > >> > > >> > > > >> > > LB outside of the VR sounds like a good idea. An >> appliance >> > > >> based >> > > >> > > on, >> > > >> > > > >> say >> > > >> > > > >> > > cloud-init + ansible and so on could do the trick; alas >> > > it'd >> > > >> need >> > > >> > > to be >> > > >> > > > >> > > outside ACS. >> > > >> > > > >> > > I guess as users we could maybe come up with a spec >> for an >> > > >> > > > >> improvement, at >> > > >> > > > >> > > least we'd have something the devs could look at >> whenever >> > > it >> > > >> is >> > > >> > > > >> possible. >> > > >> > > > >> > > >> > > >> > > > >> > > Regards, >> > > >> > > > >> > > Lucian >> > > >> > > > >> > > >> > > >> > > > >> > > -- >> > > >> > > > >> > > Sent from the Delta quadrant using Borg technology! >> > > >> > > > >> > > >> > > >> > > > >> > > Nux! >> > > >> > > > >> > > www.nux.ro >> > > >> > > > >> > > >> > > >> > > > >> > > ----- Original Message ----- >> > > >> > > > >> > > > From: "Andrija Panic" <andrija.pa...@gmail.com> >> > > >> > > > >> > > > To: "dev" <d...@cloudstack.apache.org> >> > > >> > > > >> > > > Cc: "users" <users@cloudstack.apache.org> >> > > >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37 >> > > >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for >> > > >> > > > >> > > >> > > >> > > > >> > > > We used to make some special stuff for one of the >> > > clients, >> > > >> > > where all >> > > >> > > > >> LB >> > > >> > > > >> > > > configuration work is done from outside of the ACS, >> i.e. >> > > >> python >> > > >> > > > >> script to >> > > >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for >> > > >> transparent >> > > >> > > > >> proxy, >> > > >> > > > >> > > > since client insisted on SSL termination done on >> backend >> > > >> web SSL >> > > >> > > > >> > > servers.... >> > > >> > > > >> > > > Not good idea, that is all I can say (custom >> > > configuration >> > > >> > > thing) - >> > > >> > > > >> but >> > > >> > > > >> > > the >> > > >> > > > >> > > > LB setup is actually good - transparent mode haproxy, >> > > works >> > > >> on >> > > >> > > TCP >> > > >> > > > >> level, >> > > >> > > > >> > > > so you can see "real client IP" on the backend >> servers >> > > >> (which >> > > >> > > must >> > > >> > > > >> use VR >> > > >> > > > >> > > > as the default gtw, as per default, so the whole >> setup >> > > works >> > > >> > > > >> properly). >> > > >> > > > >> > > > >> > > >> > > > >> > > > I'm still looking forward to see some special >> support of >> > > LB >> > > >> > > inside >> > > >> > > > >> VR via >> > > >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API - i.e. >> to >> > > >> enable LB >> > > >> > > > >> > > > provisioning SCRIPT (bash, or whatever), where all >> > > needed >> > > >> > > > >> > > > install+configure can be done from client side - >> > > otherwise >> > > >> > > covering >> > > >> > > > >> all >> > > >> > > > >> > > > user cases, with proper HTTP checks and similar....is >> > > >> > > impossible to >> > > >> > > > >> do >> > > >> > > > >> > > > IMHO. >> > > >> > > > >> > > > >> > > >> > > > >> > > > Some other clients, actually have internal FW >> appliance >> > > >> (i.e. >> > > >> > > > >> multihomed >> > > >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and >> > > haproxy >> > > >> > > instaled >> > > >> > > > >> on >> > > >> > > > >> > > > this device (with NAT configured from VR to this >> internal >> > > >> > > FW/VM, so >> > > >> > > > >> > > remote >> > > >> > > > >> > > > IP can be seen properly) - this setup is fully under >> > > >> customer >> > > >> > > > >> control, >> > > >> > > > >> > > and >> > > >> > > > >> > > > can provide any kind of special haproxy config... >> > > >> > > > >> > > > >> > > >> > > > >> > > > >> > > >> > > > >> > > > >> > > >> > > > >> > > > >> > > >> > > > >> > > > >> > > >> > > > >> > > > >> > > >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <n...@li.nux.ro> >> wrote: >> > > >> > > > >> > > > >> > > >> > > > >> > > >> Hello, >> > > >> > > > >> > > >> >> > > >> > > > >> > > >> Of the people running an LB (VR) with https >> backends, >> > > how >> > > >> do >> > > >> > > you >> > > >> > > > >> deal >> > > >> > > > >> > > with >> > > >> > > > >> > > >> the lack of x-forwarded-for since for port 443 >> there's >> > > just >> > > >> > > simple >> > > >> > > > >> TCP >> > > >> > > > >> > > >> balancing? >> > > >> > > > >> > > >> >> > > >> > > > >> > > >> Has anyone thought of terminating SSL in the VR >> instead? >> > > >> Ideas? >> > > >> > > > >> > > >> >> > > >> > > > >> > > >> Cheers >> > > >> > > > >> > > >> >> > > >> > > > >> > > >> -- >> > > >> > > > >> > > >> Sent from the Delta quadrant using Borg technology! >> > > >> > > > >> > > >> >> > > >> > > > >> > > >> Nux! >> > > >> > > > >> > > >> www.nux.ro >> > > >> > > > >> > > >> >> > > >> > > > >> > > > >> > > >> > > > >> > > > >> > > >> > > > >> > > > >> > > >> > > > >> > > > -- >> > > >> > > > >> > > > >> > > >> > > > >> > > > Andrija Panić >> > > >> > > > >> > > >> > > >> > > > >> >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > -- >> > > >> > > > > >> > > >> > > > > Andrija Panić >> > > >> > > >> > >
