Hi Aleks, Yes, I should have done last in my last email post. Sorry about that.
haproxy version: [root@crsplabnet2 haproxy]# haproxy -vv HA-Proxy version 1.8.14-52e4d43 2018/09/20 Copyright 2000-2018 Willy Tarreau <wi...@haproxy.org> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017 Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Encrypted password support via crypt(3): yes Built with multi-threading support. Built with PCRE version : 8.32 2012-11-30 Running on PCRE version : 8.32 2012-11-30 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with zlib version : 1.2.7 Running on zlib version : 1.2.7 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with network namespace support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available filters : [SPOE] spoe [COMP] compression [TRACE] trace ----------------------------- Shibboleth version ( running in both of my backend nodes, as HAproxy can't interact with shib , to do authentication offload in HAProxy ) [root@web1 ~]# rpm -qa |grep hibb shibboleth-3.0.2-1.1.x86_64 ------------------------------- Apache version ( running in both backend nodes ONLY ) : [root@web1 ~]# rpm -qa |grep httpd httpd-devel-2.4.6-80.el7.centos.x86_64 httpd-tools-2.4.6-80.el7.centos.x86_64 httpd-2.4.6-80.el7.centos.x86_64 ----------------------------------- Linux version in all systems: Centos 7.5 , running stock kernel 3.10.0-862.el7.x86_64 here is a workflow and some background info: background info: A. the application that we are trying to use: jupyterhub -- https://github.com/jupyterhub B. We have shibboleth IDP that all our internal sites do authentication from. C. This application Jupyterhub spawns its own ( internally built ) proxy instance and binds that to port 8000, as you saw in the <Location> config section in apache config above. D. Jupyterhub can directly authenticate with shibboleth ( some additional work is needed ), but since we are putting this behind HAProxy, running Shibboleth SP in the backend node ( where the jupyterhub instance runs ) is needed. therefore, I decided to wrap jupytrehub within apache as apache has built-in support for shibboleth ( mod_shib , that gets installed with shibboleth RPM ). Here is what I was thinking expected workflow would be: Client [ public IP ] connects ---> HAproxy [ public IP ] connects ---> backend node [ private IP:80 ] Note: Shibboleth is running in the backend node with its public IP: 443, as it loves to work with SSL and I needed an in-common SSL cert installed to get the SP working correctly. So, for example, in the browser, I go to: -- proxy.example.com ( HAProxy system ) -- from the backend, shibboleth responds to me with a login screen. -- I authenticate with username/password -- Shibboleth hands my username as an attribute called "UserID". to Apache, and I forward that "UserID" to jupyterhub. Apache pulls the jupyterhub instance, presents to the user. Here is what I expect: Type: URL in the browser: 'proxy.exmaple.com/jhub' After authentication, the browse URL shown to connect directly to the backend node, such as: https://backendnode1.example.com/jhub/itoufiqu/tree? whereas, I would expect it to keep the proxy URL to be: https://proxy.exmaple.com/jhub/itoufiqu/tree? Interesting thing is, this redirect to the backend machine, connects the client directly to the backend machine over the public IP of the backend machine. I think this is due to SSO being done by shibboleth. Since Shibboleth fails authentication from within the internal LAN ( as the IDP could not talk to the SP in the private network), I had to get a public IP. In this process, HAProxy loses all accounting data to this session, except the first connection attempt. This setup could be good or bad, I am not sure. My initial thoughts were, all traffic passes through HAProxy. In retrospect, HAProxy is not being taxed at all, there is a direct connection between the client and the backend node over a public IP ( of the backend node). But, what I don't understand is, if a connection attempt is made through HAProxy, why would it allow the connection to be handed off to the backend node directly ( client -> backend node )? Thoughts? On Sat, Oct 27, 2018 at 4:31 AM Aleksandar Lazic <al-hapr...@none.at> wrote: > Hi Imam. > > It would be helpfull to know your used versions: > > haproxy -vv > apache httpd version > shibboleth version > > A small workflow picture like: > > Client -> haproxy -> apache httpd -> shibboleth ? > > Am 27.10.2018 um 07:44 schrieb Imam Toufique: > > Hi Igor, > > > > Thanks very much for offering to help! I will do this in sections, > hopefully, I > > can keep this from being too cluttered. > > > > haproxy.cfg: > > > -------------------------------------------------------------------------------------- > > global > > #log /dev/log local0 debug > > #log /dev/log local1 debug > > log 127.0.0.1 local2 > > chroot /var/lib/haproxy > > stats timeout 30s > > user haproxy > > group haproxy > > tune.ssl.default-dh-param 2048 > > daemon > > > > defaults > > log global > > mode http > > option tcplog > > option dontlognull > > timeout connect 5000 > > timeout client 50000 > > timeout server 50000 > > timeout tunnel 9h > > option tcp-check > > > > frontend http_front > > bind :80 > > bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem > > stats uri /haproxy?stats > > default_backend web1_cluster > > option httplog > > log global > > #option dontlognull > > log /dev/log local0 debug > > The 2 log entries are redundant, imho. > > > mode http > > Is set at default block. Please take a look at > > https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/ > > > option forwardfor # forward IP > > http-request set-header X-Forwarded-Port %[dst_port] > > http-request add-header X-Forwarded-Proto https if { ssl_fc } > > I personally would use here also set-header instead of add. > > > redirect scheme https if !{ ssl_fc } > > > > acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav > > This should be only the host name. > > https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.6-hdr > and > https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.6-req.hdr > > I would add a addition acl. > > acl host_web2 hdr(host) -i crsplab2.oit.uci.edu > acl path_web2 path_beg -i /webdav > > > use_backend webdav_cluster if host_web2 > > an add it to the use_backend line > > use_backend webdav_cluster if host_web2 path_web2 > > > acl host_web3 path_beg /jhub > > use_backend web3_cluster if host_web3 > > > > > > backend webdav_cluster > > balance roundrobin > > I would add here the following line. > > cookie SRV1 insert indirect nocache > > server web1 10.1.100.156:8080 check inter 2000 cookie w1 > > server web2 10.1.100.160:8080 check inter 2000 cookie w2 > > > > backend web3_cluster > > I would add here the following line. > > cookie SRV2 insert indirect nocache > > server publicIP:443 check ssl verify none inter 2000 cookie w1 > > > ----------------------------------------------------------------------------------------------------- > > Note: I have a single backend node, as it was easy to test with just one > node, > > instead of making changes to 2 nodes at a time. > > > > Here is my apache config: > > > > in httpd.conf, only change I have made is ( the rest is a stock centos > 7.5 > > httpd.conf ): > > ------------------------------------- > > ServerName 10.1.100.160:80 ( Internal IP of the backend node) > > Redirect permanent /jhub https://crsplabweb1.domain.com/jhub > > ------------------------------------- > > > > in my ssl.conf, where I access the jupyterhub instance running in > 127.0.0.1:8000 > > <http://127.0.0.1:8000> . Also, note that the backend is running > shibboleth > > SP. One of the issues I encountered is, If I did not have SSL , i was > getting a > > browser warning for not having SSL. > Can you set up shibboleth in that manner that he answers with > proxy.domain.com? > As we don't know which version is in place I send you just as a hint here > some > links. > > https://wiki.shibboleth.net/confluence/display/SHIB2/SPReverseProxy > > http://shibboleth.1660669.n2.nabble.com/shibboleth-sp-behind-an-HAproxy-td5960149.html > > https://duckduckgo.com/?q=shibboleth+behind+haproxy > > > > Here is my ssl.conf: > > > > > -------------------------------------------------------------------------- > > Listen 443 https > > SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog > > SSLSessionCache shmcb:/run/httpd/sslcache(512000) > > SSLSessionCacheTimeout 300 > > SSLRandomSeed startup file:/dev/urandom 256 > > SSLRandomSeed connect builtin > > SSLCryptoDevice builtin > > > > <VirtualHost _default_:443> > > > > UseCanonicalName on > > ServerName crsplabweb1.domain.com:443 > > Maybe you can change this to proxy.domain.com and add a ServerAlias > > ServerName proxy.domain.com > ServerAlias crsplabweb1 crsplabweb1.domain.com > > https://httpd.apache.org/docs/2.4/mod/core.html#serveralias > > > ErrorLog logs/ssl_error_log > > TransferLog logs/ssl_access_log > > LogLevel warn > > > > SSLEngine on > > > > SSLProtocol all -SSLv2 -SSLv3 > > SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA > > SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer > > SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key > > SSLCertificateChainFile > /etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c > > > > <Files ~ "\.(cgi|shtml|phtml|php3?)$"> > > SSLOptions +StdEnvVars > > </Files> > > <Directory "/var/www/cgi-bin"> > > SSLOptions +StdEnvVars > > </Directory> > > > > <Location /jhub> > > ProxyPass http://127.0.0.1:8000/jhub > > ProxyPassReverse http://127.0.0.1:8000/jhub > > RequestHeader unset Accept-Encoding > > ProxyPreserveHost on > > AuthType shibboleth > > ShibRequestSetting requireSession 1 > > Require shibboleth > > ShibUseHeaders On > > ShibBasicHijack On > > RewriteEngine On > > RequestHeader set X-Remote-User %{REMOTE_USER}s > > </Location> > > > > <LocationMatch > "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> > > ProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3 > > ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3 > > </LocationMatch> > > > > BrowserMatch "MSIE [2-5]" \ > > nokeepalive ssl-unclean-shutdown \ > > downgrade-1.0 force-response-1.0 > > > > CustomLog logs/ssl_request_log \ > > "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" > > </VirtualHost> > > > ---------------------------------------------------------------------------------- > > > > Thanks > > Regards > Aleks > > > On Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov < > ig...@encompasscorporation.com > > <mailto:ig...@encompasscorporation.com>> wrote: > > > > Hi Imam, > > > > On Sat, Oct 27, 2018 at 9:37 AM Imam Toufique <techie...@gmail.com > > <mailto:techie...@gmail.com>> wrote: > > > > Hi, > > > > I came up with the following config, things seem to be working > now, for > > the most part. > > > > frontend http_front > > bind :80 > > bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem > > stats uri /haproxy?stats > > default_backend web1_cluster > > option httplog > > log global > > #option dontlognull > > log /dev/log local0 debug > > mode http > > option forwardfor # forward IP > > http-request set-header X-Forwarded-Port %[dst_port] > > http-request add-header X-Forwarded-Proto https if { ssl_fc } > > redirect scheme https if !{ ssl_fc } > > acl host_web3 path_beg /jhub > > use_backend web3_cluster if host_web3 > > > > web3_cluster > > > > backend web3_cluster > > mode http > > balance source > > server crsplabweb1.domain.com publicIP:443 check ssl verify > none inter 2000 cookie w1 > > > > The above config gets me to the backend node -- where I have a > > jupyterhub instance running + . Shibboleth SP running for > > authentication. As I could not get shibboleth SP to work by > staying in > > my private network, I had to set up a public IP for the backend > node, > > get SSL certs - so shibboleth authentication could be done. I > am sure > > there is a better approach to this, but I don't know what it > is. I will > > be trying out SNAT to see if that will allow me to keep using my > private > > IP for the backend nodes. If any of you know how to do SNAT, > please > > chime in, it would be worth the time/effort to try it out. > > > > Now, the interesting thing I have noticed with the above setup > -- when I > > connect to HAProxy, let's say with https://proxy.domain.com , I > > authenticate with shibboleth, and then the URL in the browser > points to > > the backend node. > > > > For example: > > > > my proxy address: https://proxy.domain.com/jhub > > > > after I connect to the backend, the URL turns into - > > https://crsplabweb1.domain.com/jhub/tree? > > > > ...and everything works thereafter. > > > > I tried the rewrite method that Igor has suggested before, that > did not > > make any difference. But what I noticed is, after I connect, no > traffic > > go through the proxy anymore, my client ( i.e. laptop) connects > directly > > to the backend server. Not sure if this good or bad though (?) , > but, I > > am not sure how to configure this so that I will go through a > proxy but > > still be connected in the backend via a private IP and I can ( > still ) > > authenticate via shibboleth. > > > > So, when I change the 'web3_cluster' backend to : > > > > server crsplabweb1 privateIP:80 inter 2000 cookie w1 > > > > and, I set backend apache to accept connection on port 80, then > I break > > shibboleth authentication. > > > > Any inputs here? > > > > thanks, guys! > > > > > > I think it is time for you to provide the full HAP and Apache > configs so we > > can see what is going on (please obfuscate any sensitive data). Also > the use > > of the "cookie w1" is not clear since you are not setting it in HAP > and is > > kinda redundant for single backend setup. > > > > > > > > On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov > > <ig...@encompasscorporation.com <mailto: > ig...@encompasscorporation.com>> > > wrote: > > > > > > > > On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov > > <ig...@encompasscorporation.com > > <mailto:ig...@encompasscorporation.com>> wrote: > > > > > > > > On Thu, 25 Oct 2018 6:13 pm Imam Toufique < > techie...@gmail.com > > <mailto:techie...@gmail.com>> wrote: > > > > so I almost got this to work, based on the situation > I am > > in. To elaborate just a bit, my setup involves a > shibboleth > > SP that I need to authenticate my application. > Since I > > can't set up the HA proxy node with shibboleth SP - > I had to > > wrap my application in the backend with apache so I > can pass > > REMOTE_USER to the application. the application I > have is - > > jupyterhub and it start with its own proxy. Long > story > > short, here is my current setup: > > > > frontend > > bind :80 > > bind :443 ssl crt /etc/haproxy/crsplab2_1.pem > > stats uri /haproxy?stats > > default_backend web1_cluster > > option httplog > > log global > > #option dontlognull > > log /dev/log local0 debug > > mode http > > option forwardfor # forward IP > > http-request set-header X-Forwarded-Port > %[dst_port] > > http-request add-header X-Forwarded-Proto https > if { ssl_fc } > > redirect scheme https if !{ ssl_fc } > > > > acl host_web3 path_beg /jhub > > use_backend web3_cluster if host_web3 > > > > backend > > server web1.oit.uci.edu <http://web1.oit.uci.edu> > > 128.110.80.5:80 <http://128.110.80.5:80> check > > > > this works for the most part. But I am confused > with a > > problem. when I get to my application, my backend IP > address > > shows up in the browser URL. > > > > for example, I see this in my browser: > > > > http://128.110.80.5/jhub/user/itoufiqu/tree? > > > > whereas, I was expecting that it would show the > original > > URL, such as: > > > > http://crsplab2.domain.com/jhub/user/itoufiqu/tree > > <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>? > ( > > where crsplab2.domain.com < > http://crsplab2.domain.com> is > > the URL to get HAproxy ) > > > > > > You need to tell your backend app that it runs behind > reverse > > proxy with ssl termination and that it's domain/url > > is https://crsplab2.domain.com > > <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>. > How you do > > that depends on the backend app you are using but most > of them > > like apache2, tomcat etc. have specific configs that you > can > > find in their documentation. For example if your backend > is > > apache2 I bet you don't have the DomainName set in the > config in > > which case it defaults to the host ip address. > > > > > > You can also try: > > > > rspirep ^Location:\ http://(.*):80(.*) Location:\ > > https://crsplab2.domain.com > > <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2 > if { > > ssl_fc } > > > > to fix the URL but note that this will not save you from > hard coded > > url's in the returned html pages the way apache does. > > > > > > > > While I am no expert in HA proxy world, I think this > might > > due to the fact that my backend does not have SSL and > > HAproxy frontend does have SSL. At this point, I > would > > avoid that IP address showing up in the browser. > what is > > the best way to accomplish this? > > > > thanks for your continues help! > > > > > > > > > > > > > > On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic > > <al-hapr...@none.at <mailto:al-hapr...@none.at>> > wrote: > > > > Hi. > > > > Am 23.10.2018 um 09:04 schrieb Imam Toufique: > > > I am looking for some help on how to write the > > following apache proxypass rules > > > in HAproxy. Not to mention I am at a bit of > loss with > > my first try :-) . Here > > > are my current proxypass rules: > > > > > > ProxyPass http://10.1.100.156:8000/jhub > > > ProxyPassReverse http://10.1.100.156:8000/jhub > > > > Well ProxyPass and ProxyPassReverse do a lot of > thinks > > not just rewrites, as > > mentioned in the doc > > > > > https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass > > > https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse > > > > > > > <LocationMatch > > > "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> > > > ProxyPassMatch ws:// > 10.1.100.156:8000/jhub/$1/$2$3 > > <http://10.1.100.156:8000/jhub/$1/$2$3> > > > ProxyPassReverse > > ws://10.1.100.156:8000/jhub/$1/$2$3 > > <http://10.1.100.156:8000/jhub/$1/$2$3> > > > </LocationMatch> > > > > > > As I am not well versed in the massive HAproxy > > configuration guide, if any of > > > you can give me a hand with this, I would > very much > > appreciate it. > > > > I'm also not "that" expert but I would try the > > following, untested. > > > > ### > > defaults > > mode http > > log global > > > > #... maybe some other settings > > timeout tunnel 10h > > > > frontend https_001 > > > > #... maybe some other settings > > > > acl websocket path_beg /jhub > > > > #... maybe some other acls > > > > use_backend websocket_001 if websocket > > > > backend websocket_001 > > > > reqrep "^([^\ :]*) > > > /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" > > "/jhub/\1/\2\3" > > > > # You will need to replace the first column > with the > > response from the > > # backend response > > # rspirep "^Location: > > > /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" > > "Location: > > /jhub/\1/\2\3" > > # OR > > # http-response replace-header Location > > > "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" > > "/jhub/\1/\2\3" > > > > # add some checks > > > > server ws_01 10.1.100.156:8000 > > <http://10.1.100.156:8000> check > > ### > > > > Here are some links which may help you also. > > > > > https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/ > > > https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep > > > https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep > > > > I would run haproxy in Debug mode and see how the > > request pass haproxy and adopt > > the config. > > > > It would be nice when you show us the working > conf ;-) > > > > It would be nice to have a > > > > http-request replace-uri <match-regex> > <replace-fmt> > > > > to replace the reqrep. > > > -- Regards, *Imam Toufique* *213-700-5485*