I am still trying to fight this battle...
This is the command I am running on Exchange to put it into maintenance mode:
Set-ServerComponentState $nodeDOWN -Component HubTransport -State Draining
-Requester Maintenance
Set-ServerComponentState $nodeDOWN -Component ServerWideOffline -State
InActive -Requester Maintenance
This sets all the components to "inactive". This is important as we have a DAG
and we don't want mailboxes to failback while still doing maintenance, as well
as mitigate potential data loss.
The inactive component state "disables" the healthcheck.htm pages (I can't
think of a better way to describe what happens to the healthcheck pages than
disabled)
However, what I have now noticed is that even though the component state is
inactive, the service is still "up".
For instance;
Disable OAB site via this Exchange command:
Set-ServerComponentState <servername> -Component OabProxy -State
InActive -Requester Maintenance
Verify with command:
Get-ServerComponentState <servername>
Try to go to health check page and it does NOT load:
https://<servername>/oab/healthcheck.htm
Try to go to oab.xml file and it DOES load:
https://<servername>/oab/5453f2e4-8451-43c5-99a1-5ae1aa4ce41d/oab.xml
To recap;
Exchange says component is Inactive
IIS is up and still serving content
healthcheck.htm page does not load, is down, unavailable, what have you
haproxy gets 200 response from health check that supposedly isn't available
Here are relevant haproxy logs showing the health check as good and content
still being proxied, even though the component is inactive (ie health check
page is not accessible)
Oct 29 14:51:36 localhost haproxy[93952]: <IP-ADDRESS>:58359
[29/Oct/2020:14:51:36.345] fe_ex2019~ be_ex2019_oab/<SERVER_NAME_2> 0/0/0/34/34
206 1806 - - ---- 22/22/0/0/0 0/0 {Microsoft-IIS/10.0|text/xml} "GET
/OAB/5453f2e4-8451-43c5-99a1-5ae1aa4ce41d/oab.xml HTTP/1.1"
Oct 29 14:51:39 localhost haproxy[93952]: <IP-ADDRESS>:58359
[29/Oct/2020:14:51:39.153] fe_ex2019~ be_ex2019_oab/<SERVER_NAME_2> 0/0/0/35/35
206 2790 - - ---- 25/25/0/0/0 0/0 {Microsoft-IIS/10.0|text/xml} "GET
/OAB/5453f2e4-8451-43c5-99a1-5ae1aa4ce41d/oab.xml HTTP/1.1"
Oct 29 14:51:39 <Server-Name> haproxy: [WARNING] 302/145139 (93952) :
Health check for server be_ex2019_oab/<SERVER_NAME_1> succeeded, reason: Layer7
check passed, code: 200, info: "HTTP status check returned code <3C>200<3E>",
check duration: 8ms, status: 3/3 UP.
Oct 29 14:51:40 localhost haproxy[93952]: <IP-ADDRESS>:58359
[29/Oct/2020:14:51:40.359] fe_ex2019~ be_ex2019_oab/<SERVER_NAME_2> 0/0/0/35/35
206 3362 - - ---- 28/28/0/0/0 0/0 {Microsoft-IIS/10.0|text/xml} "GET
/OAB/5453f2e4-8451-43c5-99a1-5ae1aa4ce41d/oab.xml HTTP/1.1"
Oct 29 14:51:40 <Server-Name> haproxy: [WARNING] 302/145140 (93952) :
Health check for server be_ex2019_oab/<SERVER_NAME_2> succeeded, reason: Layer7
check passed, code: 200, info: "HTTP status check returned code <3C>200<3E>",
check duration: 11ms, status: 3/3 UP.
Oct 29 14:51:41 localhost haproxy[93952]: <IP-ADDRESS>:58359
[29/Oct/2020:14:51:41.362] fe_ex2019~ be_ex2019_oab/<SERVER_NAME_1> 0/0/0/32/32
206 8428 - - ---- 32/31/0/0/0 0/0 {Microsoft-IIS/10.0|text/xml} "GET
/OAB/5453f2e4-8451-43c5-99a1-5ae1aa4ce41d/oab.xml HTTP/1.1"
Oct 29 14:51:42 localhost haproxy[93952]: <IP-ADDRESS>:58359
[29/Oct/2020:14:51:42.398] fe_ex2019~ be_ex2019_oab/<SERVER_NAME_2> 0/0/0/36/37
206 18113 - - ---- 30/29/0/0/0 0/0 {Microsoft-IIS/10.0|text/xml} "GET
/OAB/5453f2e4-8451-43c5-99a1-5ae1aa4ce41d/oab.xml HTTP/1.1"
Looking at the IIS logs, when the component is active, I see the GET requests
from my workstations IP. When the component is inactive, no GET request is
logged from my workstation.
In addition, weather the service is active or inactive, IIS logs GET requests
from the haproxy servers:
2020-10-30 00:13:01 10.168.99.91 GET /oab/healthcheck.htm - 443 -
<haproxy_server_ip_1> - - 200 0 0 1
2020-10-30 00:13:11 10.168.99.91 GET /oab/healthcheck.htm - 443 -
<haproxy_server_ip_2> - - 200 0 0 1
2020-10-30 00:13:15 10.168.99.91 GET /oab/healthcheck.htm - 443 -
<haproxy_server_ip_1> - - 200 0 0 1
2020-10-30 00:13:25 10.168.99.91 GET /oab/healthcheck.htm - 443 -
<haproxy_server_ip_2> - - 200 0 0 1
2020-10-30 00:13:30 10.168.99.91 GET /oab/healthcheck.htm - 443 -
<haproxy_server_ip_1> - - 200 0 0 1
2020-10-30 00:13:41 10.168.99.91 GET /oab/healthcheck.htm - 443 -
<haproxy_server_ip_2> - - 200 0 0 1
If IIS is disabled, haproxy works as expected.
If the Exchange server is shutdown, haproxy works as expected.
The issue is getting haproxy to recognize when the e=Exchange server is in
maintenance mode.
I found this site and his testing is basically the same thing I am doing, yet
he is getting a proper result.
Under the "Testing" heading
http://ezoltan.blogspot.com/2014/10/highly-available-l7-load-balancing-for.html
He is using Exchange 2013 and haproxy version 1.5.4, but otherwise I don't see
how our configs differ to where he gets the proper result and I do not.
I am also have a suspicion that this was never working properly in our
implementation. So I don't think the issue I am having is attributable to any
particular version of haproxy.
Does anyone have any ideas on how I can get haproxy to recognize when Exchange
is in maintenance mode?
Its baffling to me how the health check can pass and get a 200 response when
the page isn't accessible by any other means (browsers, wget).
It seems haproxy's health check requests are fundamentally different than a
regular browser request and that this what is driving the difference in
response.
Hopefully there is a way to configure haproxy to request a health check in such
a way that an Exchange service in maintenance mode is properly detected as down.
Thanks,
-Luke
From: Wesley Lukehart
Sent: Wednesday, October 14, 2020 18:28
To: 'haproxy@formilux.org' <haproxy@formilux.org>
Subject: Heath check responds up even when server is down
Hello fine people. Short time lurker, first time poster.
Was on version 2.0.5 with CentOS 7.6 and everything was working fine with
Exchange 2019.
Upgraded to 2.2.3 and now when we put Exchange into maintenance mode HAProxy
does not change status - it reports that all services are still up (L7OK/200).
Example backend:
backend be_ex2019_oab
mode http
balance roundrobin
option httpchk GET /oab/healthcheck.htm
option log-health-checks
http-check expect status 200
server <servername> <IP_Address>:443 check ssl inter 15s verify required
ca-file <Path_to_crt_file>
server <servername> <IP_Address>:443 check ssl inter 15s verify required
ca-file <Path_to_crt_file>
If I stop the app pool for a service in IIS, or stop all of IIS, HAProxy will
properly show the service/services as down - as it gets a non 200 response (503
or 404).
When putting the Exchange server into maintenance mode, there is no http
response.
When I check with a browser I get "ERR_HTTP2_PROTOCOL_ERROR" or "Secure
Connection Failed". Basically no response.
When I check with wget from the haproxy server I get "HTTP request sent,
awaiting response... Read error (Connection reset by peer) in headers."
Yet HAProxy is happy and continues to try to send mail to the down server - not
good.
Any Ideas?
I just tried 2.2.4 and no joy.
Thanks,
-Luke
