Hi Imam,
On Sat, Oct 27, 2018 at 4:42 PM Imam Toufique <techie...@gmail.com> wrote:
> Hi Igor,
>
> Thanks very much for offering to help! I will do this in sections,
> hopefully, I can keep this from being too cluttered.
>
> haproxy.cfg:
>
> --------------------------------------------------------------------------------------
> global
> #log /dev/log local0 debug
> #log /dev/log local1 debug
> log 127.0.0.1 local2
> chroot /var/lib/haproxy
> stats timeout 30s
> user haproxy
> group haproxy
> tune.ssl.default-dh-param 2048
> daemon
>
> defaults
> log global
> mode http
> option tcplog
> option dontlognull
> timeout connect 5000
> timeout client 50000
> timeout server 50000
> timeout tunnel 9h
> option tcp-check
>
> frontend http_front
> bind :80
> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem
> stats uri /haproxy?stats
> default_backend web1_cluster
> option httplog
> log global
> #option dontlognull
> log /dev/log local0 debug
> mode http
> option forwardfor # forward IP
> http-request set-header X-Forwarded-Port %[dst_port]
> http-request add-header X-Forwarded-Proto https if { ssl_fc }
> redirect scheme https if !{ ssl_fc }
>
> acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav
> use_backend webdav_cluster if host_web2
>
> acl host_web3 path_beg /jhub
> use_backend web3_cluster if host_web3
>
>
> backend webdav_cluster
> balance roundrobin
> server web1 10.1.100.156:8080 check inter 2000 cookie w1
> server web2 10.1.100.160:8080 check inter 2000 cookie w2
>
> backend web3_cluster
> server publicIP:443 check ssl verify none inter 2000 cookie w1
>
> -----------------------------------------------------------------------------------------------------
> Note: I have a single backend node, as it was easy to test with just one
> node, instead of making changes to 2 nodes at a time.
>
> Here is my apache config:
>
> in httpd.conf, only change I have made is ( the rest is a stock centos 7.5
> httpd.conf ):
> -------------------------------------
> ServerName 10.1.100.160:80 ( Internal IP of the backend node)
> Redirect permanent /jhub https://crsplabweb1.domain.com/jhub
> -------------------------------------
>
> in my ssl.conf, where I access the jupyterhub instance running in
> 127.0.0.1:8000 . Also, note that the backend is running shibboleth SP.
> One of the issues I encountered is, If I did not have SSL , i was getting a
> browser warning for not having SSL.
>
> Here is my ssl.conf:
>
> --------------------------------------------------------------------------
> Listen 443 https
> SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
> SSLSessionCache shmcb:/run/httpd/sslcache(512000)
> SSLSessionCacheTimeout 300
> SSLRandomSeed startup file:/dev/urandom 256
> SSLRandomSeed connect builtin
> SSLCryptoDevice builtin
>
> <VirtualHost _default_:443>
>
> UseCanonicalName on
> ServerName crsplabweb1.domain.com:443
>
> ErrorLog logs/ssl_error_log
> TransferLog logs/ssl_access_log
> LogLevel warn
>
> SSLEngine on
>
> SSLProtocol all -SSLv2 -SSLv3
> SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA
> SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer
> SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key
> SSLCertificateChainFile
> /etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c
>
> <Files ~ "\.(cgi|shtml|phtml|php3?)$">
> SSLOptions +StdEnvVars
> </Files>
> <Directory "/var/www/cgi-bin">
> SSLOptions +StdEnvVars
> </Directory>
>
> <Location /jhub>
> ProxyPass http://127.0.0.1:8000/jhub
> ProxyPassReverse http://127.0.0.1:8000/jhub
> RequestHeader unset Accept-Encoding
> ProxyPreserveHost on
> AuthType shibboleth
> ShibRequestSetting requireSession 1
> Require shibboleth
> ShibUseHeaders On
> ShibBasicHijack On
> RewriteEngine On
> RequestHeader set X-Remote-User %{REMOTE_USER}s
> </Location>
>
> <LocationMatch
> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)">
> ProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3
> ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3
> </LocationMatch>
>
> BrowserMatch "MSIE [2-5]" \
> nokeepalive ssl-unclean-shutdown \
> downgrade-1.0 force-response-1.0
>
> CustomLog logs/ssl_request_log \
> "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
> </VirtualHost>
>
> ----------------------------------------------------------------------------------
>
> Thanks
>
Your problem is that you are not using the Forwarded headers set by HAP in
Apache thus you get http response instead ssl.
First for haproxy create a directory where you will keep all your SSL
certs, lets say /etc/haproxy/ssl.d/, and put the crsplab2.oit.uci.edu and
crsplabweb1.domain.com certificates inside. More details on setting SSL
certificates in Haproxy can be found here:
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crt
The config will then look something like this:
frontend http_front
bind *:80
bind *:443 ssl crt /etc/haproxy/ssl.d/ no-sslv3 no-tls-tickets ...
backend web3_cluster
server shibboleth1 10.1.100.160:80 check inter 2000
On the apache side remove the ssl settings (since now HAP will be
terminating SSL) and set a SSL redirect, something like this:
<VirtualHost *:80>
ServerName crsplabweb1.domain.com
ServerAlias www.crsplabweb1.domain.com
SetEnvIfNoCase X-Forwarded-Proto https HTTPS=on
# Insure the pages requested over ssl are always over ssl
RewriteEngine On
RewriteCond %{HTTP_X_Forwarded_Proto} ^https$
RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R,L]
...
</VirtualHost>
Let me know if any further questions.
> On Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov <
> ig...@encompasscorporation.com> wrote:
>
>> Hi Imam,
>>
>> On Sat, Oct 27, 2018 at 9:37 AM Imam Toufique <techie...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I came up with the following config, things seem to be working now, for
>>> the most part.
>>>
>>> frontend http_front
>>> bind :80
>>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem
>>> stats uri /haproxy?stats
>>> default_backend web1_cluster
>>> option httplog
>>> log global
>>> #option dontlognull
>>> log /dev/log local0 debug
>>> mode http
>>> option forwardfor # forward IP
>>> http-request set-header X-Forwarded-Port %[dst_port]
>>> http-request add-header X-Forwarded-Proto https if { ssl_fc }
>>> redirect scheme https if !{ ssl_fc }
>>> acl host_web3 path_beg /jhub
>>> use_backend web3_cluster if host_web3
>>>
>>> web3_cluster
>>>
>>> backend web3_cluster
>>> mode http
>>> balance source
>>> server crsplabweb1.domain.com publicIP:443 check ssl verify none
>>> inter 2000 cookie w1
>>>
>>> The above config gets me to the backend node -- where I have a
>>> jupyterhub instance running + . Shibboleth SP running for authentication.
>>> As I could not get shibboleth SP to work by staying in my private network,
>>> I had to set up a public IP for the backend node, get SSL certs - so
>>> shibboleth authentication could be done. I am sure there is a better
>>> approach to this, but I don't know what it is. I will be trying out SNAT
>>> to see if that will allow me to keep using my private IP for the backend
>>> nodes. If any of you know how to do SNAT, please chime in, it would be
>>> worth the time/effort to try it out.
>>>
>>> Now, the interesting thing I have noticed with the above setup -- when I
>>> connect to HAProxy, let's say with https://proxy.domain.com , I
>>> authenticate with shibboleth, and then the URL in the browser points to the
>>> backend node.
>>>
>>> For example:
>>>
>>> my proxy address: https://proxy.domain.com/jhub
>>>
>>> after I connect to the backend, the URL turns into -
>>> https://crsplabweb1.domain.com/jhub/tree?
>>>
>>> ...and everything works thereafter.
>>>
>>> I tried the rewrite method that Igor has suggested before, that did not
>>> make any difference. But what I noticed is, after I connect, no traffic go
>>> through the proxy anymore, my client ( i.e. laptop) connects directly to
>>> the backend server. Not sure if this good or bad though (?) , but, I am not
>>> sure how to configure this so that I will go through a proxy but still be
>>> connected in the backend via a private IP and I can ( still ) authenticate
>>> via shibboleth.
>>>
>>> So, when I change the 'web3_cluster' backend to :
>>>
>>> server crsplabweb1 privateIP:80 inter 2000 cookie w1
>>>
>>> and, I set backend apache to accept connection on port 80, then I break
>>> shibboleth authentication.
>>>
>>> Any inputs here?
>>>
>>> thanks, guys!
>>>
>>>
>> I think it is time for you to provide the full HAP and Apache configs so
>> we can see what is going on (please obfuscate any sensitive data). Also the
>> use of the "cookie w1" is not clear since you are not setting it in HAP
>> and is kinda redundant for single backend setup.
>>
>>
>>>
>>> On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov <
>>> ig...@encompasscorporation.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov <
>>>> ig...@encompasscorporation.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, 25 Oct 2018 6:13 pm Imam Toufique <techie...@gmail.com> wrote:
>>>>>
>>>>>> so I almost got this to work, based on the situation I am in. To
>>>>>> elaborate just a bit, my setup involves a shibboleth SP that I need to
>>>>>> authenticate my application. Since I can't set up the HA proxy node with
>>>>>> shibboleth SP - I had to wrap my application in the backend with apache
>>>>>> so
>>>>>> I can pass REMOTE_USER to the application. the application I have is -
>>>>>> jupyterhub and it start with its own proxy. Long story short, here is my
>>>>>> current setup:
>>>>>>
>>>>>> frontend
>>>>>> bind :80
>>>>>> bind :443 ssl crt /etc/haproxy/crsplab2_1.pem
>>>>>> stats uri /haproxy?stats
>>>>>> default_backend web1_cluster
>>>>>> option httplog
>>>>>> log global
>>>>>> #option dontlognull
>>>>>> log /dev/log local0 debug
>>>>>> mode http
>>>>>> option forwardfor # forward IP
>>>>>> http-request set-header X-Forwarded-Port %[dst_port]
>>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc }
>>>>>> redirect scheme https if !{ ssl_fc }
>>>>>>
>>>>>> acl host_web3 path_beg /jhub
>>>>>> use_backend web3_cluster if host_web3
>>>>>>
>>>>>> backend
>>>>>> server web1.oit.uci.edu 128.110.80.5:80 check
>>>>>>
>>>>>> this works for the most part. But I am confused with a problem. when
>>>>>> I get to my application, my backend IP address shows up in the browser
>>>>>> URL.
>>>>>>
>>>>>> for example, I see this in my browser:
>>>>>>
>>>>>> http://128.110.80.5/jhub/user/itoufiqu/tree?
>>>>>>
>>>>>> whereas, I was expecting that it would show the original URL, such
>>>>>> as:
>>>>>>
>>>>>> http://crsplab2.domain.com/jhub/user/itoufiqu/tree? ( where
>>>>>> crsplab2.domain.com is the URL to get HAproxy )
>>>>>>
>>>>>
>>>>> You need to tell your backend app that it runs behind reverse proxy
>>>>> with ssl termination and that it's domain/url is
>>>>> https://crsplab2.domain.com
>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>. How you do that
>>>>> depends on the backend app you are using but most of them like apache2,
>>>>> tomcat etc. have specific configs that you can find in their
>>>>> documentation.
>>>>> For example if your backend is apache2 I bet you don't have the DomainName
>>>>> set in the config in which case it defaults to the host ip address.
>>>>>
>>>>
>>>> You can also try:
>>>>
>>>> rspirep ^Location:\ http://(.*):80(.*) Location:\ https://
>>>> crsplab2.domain.com
>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2 if {
>>>> ssl_fc }
>>>>
>>>> to fix the URL but note that this will not save you from hard coded
>>>> url's in the returned html pages the way apache does.
>>>>
>>>>
>>>>>
>>>>>> While I am no expert in HA proxy world, I think this might due to the
>>>>>> fact that my backend does not have SSL and HAproxy frontend does have
>>>>>> SSL.
>>>>>> At this point, I would avoid that IP address showing up in the browser.
>>>>>> what is the best way to accomplish this?
>>>>>>
>>>>>> thanks for your continues help!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic <al-hapr...@none.at>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>> Am 23.10.2018 um 09:04 schrieb Imam Toufique:
>>>>>>> > I am looking for some help on how to write the following apache
>>>>>>> proxypass rules
>>>>>>> > in HAproxy. Not to mention I am at a bit of loss with my first
>>>>>>> try :-) . Here
>>>>>>> > are my current proxypass rules:
>>>>>>> >
>>>>>>> > ProxyPass http://10.1.100.156:8000/jhub
>>>>>>> > ProxyPassReverse http://10.1.100.156:8000/jhub
>>>>>>>
>>>>>>> Well ProxyPass and ProxyPassReverse do a lot of thinks not just
>>>>>>> rewrites, as
>>>>>>> mentioned in the doc
>>>>>>>
>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass
>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse
>>>>>>>
>>>>>>>
>>>>>>> > <LocationMatch
>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)">
>>>>>>> > ProxyPassMatch ws://10.1.100.156:8000/jhub/$1/$2$3
>>>>>>> > ProxyPassReverse ws://10.1.100.156:8000/jhub/$1/$2$3
>>>>>>> > </LocationMatch>
>>>>>>> >
>>>>>>> > As I am not well versed in the massive HAproxy configuration
>>>>>>> guide, if any of
>>>>>>> > you can give me a hand with this, I would very much appreciate it.
>>>>>>>
>>>>>>> I'm also not "that" expert but I would try the following, untested.
>>>>>>>
>>>>>>> ###
>>>>>>> defaults
>>>>>>> mode http
>>>>>>> log global
>>>>>>>
>>>>>>> #... maybe some other settings
>>>>>>> timeout tunnel 10h
>>>>>>>
>>>>>>> frontend https_001
>>>>>>>
>>>>>>> #... maybe some other settings
>>>>>>>
>>>>>>> acl websocket path_beg /jhub
>>>>>>>
>>>>>>> #... maybe some other acls
>>>>>>>
>>>>>>> use_backend websocket_001 if websocket
>>>>>>>
>>>>>>> backend websocket_001
>>>>>>>
>>>>>>> reqrep "^([^\ :]*)
>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"
>>>>>>> "/jhub/\1/\2\3"
>>>>>>>
>>>>>>> # You will need to replace the first column with the response from
>>>>>>> the
>>>>>>> # backend response
>>>>>>> # rspirep "^Location:
>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"
>>>>>>> "Location:
>>>>>>> /jhub/\1/\2\3"
>>>>>>> # OR
>>>>>>> # http-response replace-header Location
>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"
>>>>>>> "/jhub/\1/\2\3"
>>>>>>>
>>>>>>> # add some checks
>>>>>>>
>>>>>>> server ws_01 10.1.100.156:8000 check
>>>>>>> ###
>>>>>>>
>>>>>>> Here are some links which may help you also.
>>>>>>>
>>>>>>> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/
>>>>>>>
>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep
>>>>>>>
>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep
>>>>>>>
>>>>>>> I would run haproxy in Debug mode and see how the request pass
>>>>>>> haproxy and adopt
>>>>>>> the config.
>>>>>>>
>>>>>>> It would be nice when you show us the working conf ;-)
>>>>>>>
>>>>>>> It would be nice to have a
>>>>>>>
>>>>>>> http-request replace-uri <match-regex> <replace-fmt>
>>>>>>>
>>>>>>> to replace the reqrep.
>>>>>>>
>>>>>>> > thanks
>>>>>>>
>>>>>>> Hth
>>>>>>> Aleks
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> *Imam Toufique*
>>>>>> *213-700-5485*
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Igor Cicimov | DevOps
>>>>
>>>>
>>>> p. +61 (0) 433 078 728
>>>> e. ig...@encompasscorporation.com <http://encompasscorporation.com/>
>>>> w*.* www.encompasscorporation.com
>>>> a. Level 4, 65 York Street, Sydney 2000
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> *Imam Toufique*
>>> *213-700-5485*
>>>
>>
>>
>> --
>> Igor Cicimov | DevOps
>>
>>
>> p. +61 (0) 433 078 728
>> e. ig...@encompasscorporation.com <http://encompasscorporation.com/>
>> w*.* www.encompasscorporation.com
>> a. Level 4, 65 York Street, Sydney 2000
>>
>
>
> --
> Regards,
> *Imam Toufique*
> *213-700-5485*
>
--
Igor Cicimov | DevOps
p. +61 (0) 433 078 728
e. ig...@encompasscorporation.com <http://encompasscorporation.com/>
w*.* www.encompasscorporation.com
a. Level 4, 65 York Street, Sydney 2000
